RepoMirrors
/
haproxy
mirror of http://git.haproxy.org/git/haproxy.git/
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
haproxy/doc/SPOE.txt

1246 lines
45 KiB
Plaintext
Raw Normal View History

MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
-----------------------------------------------
Stream Processing Offload Engine (SPOE)
MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections.
2017-09-04 13:41:09 +00:00
Version 1.2
DOC: SPOE is no longer experimental The SPOE was marked as experiemental since the begining. But, it is no longer true. This can be an obstacle to its use.
2020-05-13 06:25:12 +00:00
( Last update: 2020-06-13 )
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
-----------------------------------------------
Author : Christopher Faulet
Contact : cfaulet at haproxy dot com
SUMMARY
--------
0. Terms
1. Introduction
2. SPOE configuration
2.1. SPOE scope
2.2. "spoe-agent" section
2.3. "spoe-message" section
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
2.4. "spoe-group" section
2.5. Example
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
3. SPOP specification
3.1. Data types
3.2. Frames
3.2.1. Frame capabilities
3.2.2. Frame types overview
3.2.3. Workflow
3.2.4. Frame: HAPROXY-HELLO
3.2.5. Frame: AGENT-HELLO
3.2.6. Frame: NOTIFY
3.2.7. Frame: ACK
3.2.8. Frame: HAPROXY-DISCONNECT
3.2.9. Frame: AGENT-DISCONNECT
3.3. Events & messages
3.4. Actions
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
3.5. Errors & timeouts
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
4. Logging
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
0. Terms
---------
* SPOE : Stream Processing Offload Engine.
DOC: assorted typo fixes in the documentation this is 10th iteration of typo fixes
2020-06-21 16:18:27 +00:00
A SPOE is a filter talking to servers managed by a SPOA to offload the
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
stream processing. An engine is attached to a proxy. A proxy can have
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
several engines. Each engine is linked to an agent and only one.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
* SPOA : Stream Processing Offload Agent.
A SPOA is a service that will receive info from a SPOE to offload the
stream processing. An agent manages several servers. It uses a backend to
reference all of them. By extension, these servers can also be called
agents.
* SPOP : Stream Processing Offload Protocol, used by SPOEs to talk to SPOA
servers.
This protocol is used by engines to talk to agents. It is an in-house
binary protocol described in this documentation.
1. Introduction
----------------
SPOE is a feature introduced in HAProxy 1.7. It makes possible the
communication with external components to retrieve some info. The idea started
with the problems caused by most ldap libs not working fine in event-driven
systems (often at least the connect() is blocking). So, it is hard to properly
implement Single Sign On solution (SSO) in HAProxy. The SPOE will ease this
kind of processing, or we hope so.
Now, the aim of SPOE is to allow any kind of offloading on the streams. First
DOC: SPOE is no longer experimental The SPOE was marked as experiemental since the begining. But, it is no longer true. This can be an obstacle to its use.
2020-05-13 06:25:12 +00:00
releases won't do lot of things. As we will see, there are few handled events
and even less actions supported. Actually, for now, the SPOE can offload the
processing before "tcp-request content", "tcp-response content", "http-request"
and "http-response" rules. And it only supports variables definition. But, in
spite of these limited features, we can easily imagine to implement SSO
solution, ip reputation or ip geolocation services.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
2. SPOE configuration
----------------------
Because SPOE is implemented as a filter, To use it, you must declare a "filter
spoe" line in a proxy section (frontend/backend/listen) :
frontend my-front
...
filter spoe [engine <name>] config <file>
...
The "config" parameter is mandatory. It specififies the SPOE configuration
file. The engine name is optional. It can be set to declare the scope to use in
the SPOE configuration. So it is possible to use the same SPOE configuration
for several engines. If no name is provided, the SPOE configuration must not
contain any scope directive.
We use a separate configuration file on purpose. By commenting SPOE filter
DOC: fix typos s/accidently/accidentally/ s/any ot these messages/any of theses messages/ s/catched/caught/ s/completly/completely/ s/convertor/converter/ s/desribing/describing/ s/developper/developer/ s/eventhough/even though/ s/exectution/execution/ s/functionnality/functionality/ s/If it receive a/If it receives a/ s/In can even/It can even/ s/informations/information/ s/it will be remove /it will be removed / s/langage/language/ s/mentionned/mentioned/ s/negociated/negotiated/ s/Optionnaly/Optionally/ s/ouputs/outputs/ s/outweights/outweighs/ s/ressources/resources/
2019-05-24 08:25:45 +00:00
line, you completely disable the feature, including the parsing of sections
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
reserved to SPOE. This is also a way to keep the HAProxy configuration clean.
A SPOE configuration file must contains, at least, the SPOA configuration
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
("spoe-agent" section) and SPOE messages/groups ("spoe-message" or "spoe-group"
sections) attached to this agent.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
IMPORTANT : The configuration of a SPOE filter must be located in a dedicated
file. But the backend used by a SPOA must be declared in HAProxy configuration
file.
2.1. SPOE scope
-------------------------
If you specify an engine name on the SPOE filter line, then you need to define
scope in the SPOE configuration with the same name. You can have several SPOE
scope in the same file. In each scope, you must define one and only one
"spoe-agent" section to configure the SPOA linked to your SPOE and several
DOC: assorted typo fixes in the documentation and Makefile This is another round of cleanups in various docs and comments in the Makefile.
2020-03-06 08:07:38 +00:00
"spoe-message" and "spoe-group" sections to describe, respectively, messages and
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
group of messages sent to servers mananged by your SPOA.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
A SPOE scope starts with this kind of line :
[<name>]
where <name> is the same engine name specified on the SPOE filter line. The
scope ends when the file ends or when another scope is found.
Example :
[my-first-engine]
spoe-agent my-agent
...
spoe-message msg1
...
spoe-message msg2
...
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
spoe-group grp1
...
spoe-group grp2
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
...
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
[my-second-engine]
...
If no engine name is provided on the SPOE filter line, no SPOE scope must be
found in the SPOE configuration file. All the file is considered to be in the
same anonymous and implicit scope.
MINOR: spoe: Check uniqness of SPOE engine names during config parsing The engine name is now kept in "spoe_config" struture. Because a SPOE filter can be declared without engine name, we use the SPOE agent name by default. Then, its uniqness is checked against all others SPOE engines configured for the same proxy. * TODO: Add documentation
2017-09-19 09:08:28 +00:00
The engine name must be uniq for a proxy. If no engine name is provided on the
DOC: fix a few typos in the documentation This commit deals with a few misspells in the documentation.
2018-11-14 00:55:16 +00:00
SPOE filter line, the SPOE agent name is used by default.
MINOR: spoe: Check uniqness of SPOE engine names during config parsing The engine name is now kept in "spoe_config" struture. Because a SPOE filter can be declared without engine name, we use the SPOE agent name by default. Then, its uniqness is checked against all others SPOE engines configured for the same proxy. * TODO: Add documentation
2017-09-19 09:08:28 +00:00
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
2.2. "spoe-agent" section
--------------------------
For each engine, you must define one and only one "spoe-agent" section. In this
section, you will declare SPOE messages and the backend you will use. You will
also set timeouts and options to customize your agent's behaviour.
spoe-agent <name>
Create a new SPOA with the name <name>. It must have one and only one
"spoe-agent" definition by SPOE scope.
Arguments :
<name> is the name of the agent section.
following keywords are supported :
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
- groups
MINOR: spoe: Add loggers dedicated to the SPOE agent Now it is possible to configure a logger in a spoe-agent section using a "log" line, as for a proxy. "no log", "log global" and "log <address> ..." syntaxes are supported.
2018-03-26 15:19:01 +00:00
- log
MINOR: spoe: Add "maxconnrate" and "maxerrrate" statements "maxconnrate" is the maximum number of connections per second. The SPOE will stop to open new connections if the maximum is reached and will wait to acquire an existing one. "maxerrrate" is the maximum number of errors per second. The SPOE will stop its processing if the maximum is reached. These options replace hardcoded macros MAX_NEW_SPOE_APPLETS and MAX_NEW_SPOE_APPLET_ERRS. We use it to limit SPOE activity, especially when servers are down..
2016-11-16 14:01:12 +00:00
- maxconnrate
- maxerrrate
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
- max-frame-size
MINOR: spoe: Add max-waiting-frames directive in spoe-agent configuration This is the maximum number of frames waiting for an acknowledgement on the same connection. This value is only used when the pipelinied or asynchronus exchanges between HAProxy and SPOA are enabled. By default, it is set to 20.
2018-01-25 14:32:22 +00:00
- max-waiting-frames
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
- messages
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
- [no] option async
MINOR: spoe: Add support for option dontlog-normal in the SPOE agent section It does the same than for proxies.
2018-03-26 15:20:36 +00:00
- [no] option dontlog-normal
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
- [no] option pipelining
- [no] option send-frag-payload
MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section By default, for a specific stream, when an abnormal/unexpected error occurs, the SPOE is disabled for all the transaction. So if you have several events configured, such error on an event will disabled all followings. For TCP streams, this will disable the SPOE for the whole session. For HTTP streams, this will disable it for the transaction (request and response). To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent' section. With this option, only the current event will be ignored.
2016-11-14 09:54:21 +00:00
- option continue-on-error
MINOR: spoe: add register-var-names directive in spoe-agent configuration In addition to "option force-set-var", recently added, this directive can be used to selectivelly register unknown variable names, without totally relaxing their registration during the runtime, like "option force-set-var" does. So there is no way for a malicious agent to exhaust memory by defining a too high number of variable names. In other hand, you need to enumerate all variable names. This could be painfull in some circumstances. Remember, this directive is only usefull when the variable names are not referenced anywhere in the HAProxy configuration or the SPOE one. Thanks to Etienne Carrire for his help on this part.
2017-12-22 09:00:55 +00:00
- option force-set-var
MINOR: spoe: Add "option set-on-error" statement It defines the variable to set when an error occurred during an event processing. It will only be set when an error occurred in the scope of the transaction. As for all other variables define by the SPOE, it will be prefixed. So, if your variable name is "error" and your prefix is "my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error". When set, the variable is the boolean "true". Note that if "option continue-on-error" is set, the variable is not automatically removed between events processing.
2016-11-16 14:36:19 +00:00
- option set-on-error
MINOR: spoe: Add options to store processing times in variables "set-process-time" and "set-total-time" options have been added to store processing times in the transaction scope, at each event and group processing, the current one and the total one. So it is possible to get them. TODO: documentation
2018-03-22 08:08:20 +00:00
- option set-process-time
- option set-total-time
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
- option var-prefix
MINOR: spoe: add register-var-names directive in spoe-agent configuration In addition to "option force-set-var", recently added, this directive can be used to selectivelly register unknown variable names, without totally relaxing their registration during the runtime, like "option force-set-var" does. So there is no way for a malicious agent to exhaust memory by defining a too high number of variable names. In other hand, you need to enumerate all variable names. This could be painfull in some circumstances. Remember, this directive is only usefull when the variable names are not referenced anywhere in the HAProxy configuration or the SPOE one. Thanks to Etienne Carrire for his help on this part.
2017-12-22 09:00:55 +00:00
- register-var-names
MINOR: spoe: Remove useless 'timeout ack' option To limit the time to process an event, you should set 'timeout processing' option. So 'timeout ack' option is redundant and useless.
2016-11-19 15:47:56 +00:00
- timeout hello|idle|processing
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
- use-backend
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
groups <grp-name> ...
Declare the list of SPOE groups that an agent will handle.
Arguments :
<grp-name> is the name of a SPOE group.
DOC: fix a few typos in the documentation This commit deals with a few misspells in the documentation.
2018-11-14 00:55:16 +00:00
Groups declared here must be found in the same engine scope, else an error is
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
triggered during the configuration parsing. You can have many "groups" lines.
See also: "spoe-group" section.
MINOR: spoe: Add loggers dedicated to the SPOE agent Now it is possible to configure a logger in a spoe-agent section using a "log" line, as for a proxy. "no log", "log global" and "log <address> ..." syntaxes are supported.
2018-03-26 15:19:01 +00:00
log global
log <address> [len <length>] [format <format>] <facility> [<level> [<minlevel>]]
no log
Enable per-instance logging of events and traffic.
Prefix :
no should be used when the logger list must be flushed.
See the HAProxy Configuration Manual for details about this option.
MINOR: spoe: Add "maxconnrate" and "maxerrrate" statements "maxconnrate" is the maximum number of connections per second. The SPOE will stop to open new connections if the maximum is reached and will wait to acquire an existing one. "maxerrrate" is the maximum number of errors per second. The SPOE will stop its processing if the maximum is reached. These options replace hardcoded macros MAX_NEW_SPOE_APPLETS and MAX_NEW_SPOE_APPLET_ERRS. We use it to limit SPOE activity, especially when servers are down..
2016-11-16 14:01:12 +00:00
maxconnrate <number>
Set the maximum number of connections per second to <number>. The SPOE will
stop to open new connections if the maximum is reached and will wait to
acquire an existing one. So it is important to set "timeout hello" to a
relatively small value.
maxerrrate <number>
Set the maximum number of errors per second to <number>. The SPOE will stop
its processing if the maximum is reached.
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
max-frame-size <number>
Set the maximum allowed size for frames exchanged between HAProxy and SPOA.
It must be in the range [256, tune.bufsize-4] (4 bytes are reserved for the
frame length). By default, it is set to (tune.bufsize-4).
MINOR: spoe: Add max-waiting-frames directive in spoe-agent configuration This is the maximum number of frames waiting for an acknowledgement on the same connection. This value is only used when the pipelinied or asynchronus exchanges between HAProxy and SPOA are enabled. By default, it is set to 20.
2018-01-25 14:32:22 +00:00
max-waiting-frames <number>
Set the maximum number of frames waiting for an acknowledgement on the same
connection. This value is only used when the pipelinied or asynchronus
exchanges between HAProxy and SPOA are enabled. By default, it is set to 20.
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
messages <msg-name> ...
Declare the list of SPOE messages that an agent will handle.
Arguments :
<msg-name> is the name of a SPOE message.
Messages declared here must be found in the same engine scope, else an error
is triggered during the configuration parsing. You can have many "messages"
lines.
See also: "spoe-message" section.
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
option async
no option async
Enable or disable the support of asynchronus exchanges between HAProxy and
SPOA. By default, this option is enabled.
MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section By default, for a specific stream, when an abnormal/unexpected error occurs, the SPOE is disabled for all the transaction. So if you have several events configured, such error on an event will disabled all followings. For TCP streams, this will disable the SPOE for the whole session. For HTTP streams, this will disable it for the transaction (request and response). To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent' section. With this option, only the current event will be ignored.
2016-11-14 09:54:21 +00:00
option continue-on-error
Do not stop the events processing when an error occurred on a stream.
By default, for a specific stream, when an abnormal/unexpected error occurs,
the SPOE is disabled for all the transaction. So if you have several events
DOC: assorted typo fixes in the documentation this is 10th iteration of typo fixes
2020-06-21 16:18:27 +00:00
configured, such error on an event will disabled all following. For TCP
MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section By default, for a specific stream, when an abnormal/unexpected error occurs, the SPOE is disabled for all the transaction. So if you have several events configured, such error on an event will disabled all followings. For TCP streams, this will disable the SPOE for the whole session. For HTTP streams, this will disable it for the transaction (request and response). To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent' section. With this option, only the current event will be ignored.
2016-11-14 09:54:21 +00:00
streams, this will disable the SPOE for the whole session. For HTTP streams,
this will disable it for the transaction (request and response).
When set, this option bypass this behaviour and only the current event will
be ignored.
MINOR: spoe: Add support for option dontlog-normal in the SPOE agent section It does the same than for proxies.
2018-03-26 15:20:36 +00:00
option dontlog-normal
no option dontlog-normal
Enable or disable logging of normal, successful processing.
Arguments : none
See also: "log" and section 4 about logging.
MINOR: spoe: add force-set-var option in spoe-agent configuration For security reasons, the spoe filter was only able to change values of existing variables. In specific cases (ex : with LUA code), the name of variables are unknown at the configuration parsing phase. The force-set-var option can be enabled to register all variables.
2017-12-14 09:36:40 +00:00
option force-set-var
By default, SPOE filter only register already known variables (mainly from
parsing of the configuration). If you want that haproxy trusts the agent and
registers all variables (ex: can be useful for LUA workload), activate this
option.
Caution : this option opens to a variety of attacks such as a rogue SPOA that
asks to register too many variables.
MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section By default, for a specific stream, when an abnormal/unexpected error occurs, the SPOE is disabled for all the transaction. So if you have several events configured, such error on an event will disabled all followings. For TCP streams, this will disable the SPOE for the whole session. For HTTP streams, this will disable it for the transaction (request and response). To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent' section. With this option, only the current event will be ignored.
2016-11-14 09:54:21 +00:00
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
option pipelining
no option pipelining
Enable or disable the support of pipelined exchanges between HAProxy and
SPOA. By default, this option is enabled.
option send-frag-payload
no option send-frag-payload
Enable or disable the sending of fragmented payload to SPOA. By default, this
option is enabled.
MINOR: spoe: Add "option set-on-error" statement It defines the variable to set when an error occurred during an event processing. It will only be set when an error occurred in the scope of the transaction. As for all other variables define by the SPOE, it will be prefixed. So, if your variable name is "error" and your prefix is "my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error". When set, the variable is the boolean "true". Note that if "option continue-on-error" is set, the variable is not automatically removed between events processing.
2016-11-16 14:36:19 +00:00
option set-on-error <var name>
Define the variable to set when an error occurred during an event processing.
Arguments :
<var name> is the variable name, without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
This variable will only be set when an error occurred in the scope of the
transaction. As for all other variables define by the SPOE, it will be
prefixed. So, if your variable name is "error" and your prefix is
"my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error".
MINOR: spoe: Add status code in error variable instead of hardcoded value Now, when option "set-on-error" is enabled, we set a status code representing the error occurred instead of "true". For values under 256, it represents an error coming from the engine. Below 256, it reports a SPOP error. In this case, to retrieve the right SPOP status code, you must remove 256 to this value. Here are possible values: * 1: a timeout occurred during the event processing. * 2: an error was triggered during the ressources allocation. * 255: an unknown error occurred during the event processing. * 256+N: a SPOP error occurred during the event processing.
2017-01-04 15:39:11 +00:00
When set, the variable is an integer representing the error reason. For values
under 256, it represents an error coming from the engine. Below 256, it
reports a SPOP error. In this case, to retrieve the right SPOP status code,
you must remove 256 to this value. Here are possible values:
* 1 a timeout occurred during the event processing.
DOC: fix typos s/accidently/accidentally/ s/any ot these messages/any of theses messages/ s/catched/caught/ s/completly/completely/ s/convertor/converter/ s/desribing/describing/ s/developper/developer/ s/eventhough/even though/ s/exectution/execution/ s/functionnality/functionality/ s/If it receive a/If it receives a/ s/In can even/It can even/ s/informations/information/ s/it will be remove /it will be removed / s/langage/language/ s/mentionned/mentioned/ s/negociated/negotiated/ s/Optionnaly/Optionally/ s/ouputs/outputs/ s/outweights/outweighs/ s/ressources/resources/
2019-05-24 08:25:45 +00:00
* 2 an error was triggered during the resources allocation.
MINOR: spoe: Add status code in error variable instead of hardcoded value Now, when option "set-on-error" is enabled, we set a status code representing the error occurred instead of "true". For values under 256, it represents an error coming from the engine. Below 256, it reports a SPOP error. In this case, to retrieve the right SPOP status code, you must remove 256 to this value. Here are possible values: * 1: a timeout occurred during the event processing. * 2: an error was triggered during the ressources allocation. * 255: an unknown error occurred during the event processing. * 256+N: a SPOP error occurred during the event processing.
2017-01-04 15:39:11 +00:00
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
* 3 the frame payload exceeds the frame size and it cannot be
fragmented.
* 4 the fragmentation of a payload is aborted.
MEDIUM: spoe/rules: Process "send-spoe-group" action The messages processing is done using existing functions. So here, the main task is to find the SPOE engine to use. To do so, we loop on all filter instances attached to the stream. For each, we check if it is a SPOE filter and, if yes, if its name is the one used to declare the "send-spoe-group" action. We also take care to return an error if the action processing is interrupted by HAProxy (because of a timeout or an error at the HAProxy level). This is done by checking if the flag ACT_FLAG_FINAL is set. The function spoe_send_group is the action_ptr callback ot
2017-09-22 08:20:13 +00:00
* 5 The frame processing has been interrupted by HAProxy.
MINOR: spoe: Add status code in error variable instead of hardcoded value Now, when option "set-on-error" is enabled, we set a status code representing the error occurred instead of "true". For values under 256, it represents an error coming from the engine. Below 256, it reports a SPOP error. In this case, to retrieve the right SPOP status code, you must remove 256 to this value. Here are possible values: * 1: a timeout occurred during the event processing. * 2: an error was triggered during the ressources allocation. * 255: an unknown error occurred during the event processing. * 256+N: a SPOP error occurred during the event processing.
2017-01-04 15:39:11 +00:00
* 255 an unknown error occurred during the event processing.
* 256+N a SPOP error occurred during the event processing (see section
"Errors & timeouts").
Note that if "option continue-on-error" is set, the variable is not
automatically removed between events processing.
MINOR: spoe: Add "option set-on-error" statement It defines the variable to set when an error occurred during an event processing. It will only be set when an error occurred in the scope of the transaction. As for all other variables define by the SPOE, it will be prefixed. So, if your variable name is "error" and your prefix is "my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error". When set, the variable is the boolean "true". Note that if "option continue-on-error" is set, the variable is not automatically removed between events processing.
2016-11-16 14:36:19 +00:00
See also: "option continue-on-error", "option var-prefix".
MINOR: spoe: Add options to store processing times in variables "set-process-time" and "set-total-time" options have been added to store processing times in the transaction scope, at each event and group processing, the current one and the total one. So it is possible to get them. TODO: documentation
2018-03-22 08:08:20 +00:00
option set-process-time <var name>
Define the variable to set to report the processing time of the last event or
group.
Arguments :
<var name> is the variable name, without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
This variable will be set in the scope of the transaction. As for all other
variables define by the SPOE, it will be prefixed. So, if your variable name
is "process_time" and your prefix is "my_spoe_pfx", the variable will be
"txn.my_spoe_pfx.process_time".
When set, the variable is an integer representing the delay to process the
event or the group, in milliseconds. From the stream point of view, it is the
latency added by the SPOE processing for the last handled event or group.
If several events or groups are processed for the same stream, this value
will be overrideen.
See also: "option set-total-time".
option set-total-time <var name>
Define the variable to set to report the total processing time SPOE for a
stream.
Arguments :
<var name> is the variable name, without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
This variable will be set in the scope of the transaction. As for all other
variables define by the SPOE, it will be prefixed. So, if your variable name
is "total_time" and your prefix is "my_spoe_pfx", the variable will be
"txn.my_spoe_pfx.total_time".
When set, the variable is an integer representing the sum of processing times
for a stream, in milliseconds. From the stream point of view, it is the
latency added by the SPOE processing.
If several events or groups are processed for the same stream, this value
will be updated.
See also: "option set-process-time".
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
option var-prefix <prefix>
Define the prefix used when variables are set by an agent.
Arguments :
<prefix> is the prefix used to limit the scope of variables set by an
agent.
To avoid conflict with other variables defined by HAProxy, all variables
names will be prefixed. By default, the "spoe-agent" name is used. This
option can be used to customize it.
The prefix will be added between the variable scope and its name, separated
by a '.'. It may only contain characters 'a-z', 'A-Z', '0-9', '.' and '_', as
for variables name. In HAProxy configuration, you need to use this prefix as
a part of the variables name. For example, if an agent define the variable
"myvar" in the "txn" scope, with the prefix "my_spoe_pfx", then you should
use "txn.my_spoe_pfx.myvar" name in your HAProxy configuration.
MINOR: spoe: add force-set-var option in spoe-agent configuration For security reasons, the spoe filter was only able to change values of existing variables. In specific cases (ex : with LUA code), the name of variables are unknown at the configuration parsing phase. The force-set-var option can be enabled to register all variables.
2017-12-14 09:36:40 +00:00
By default, an agent will never set new variables at runtime: It can only set
new value for existing ones. If you want a different behaviour, see
MINOR: spoe: add register-var-names directive in spoe-agent configuration In addition to "option force-set-var", recently added, this directive can be used to selectivelly register unknown variable names, without totally relaxing their registration during the runtime, like "option force-set-var" does. So there is no way for a malicious agent to exhaust memory by defining a too high number of variable names. In other hand, you need to enumerate all variable names. This could be painfull in some circumstances. Remember, this directive is only usefull when the variable names are not referenced anywhere in the HAProxy configuration or the SPOE one. Thanks to Etienne Carrire for his help on this part.
2017-12-22 09:00:55 +00:00
force-set-var option and register-var-names directive.
register-var-names <var name> ...
Register some variable names. By default, an agent will not be allowed to set
new variables at runtime. This rule can be totally relaxed by setting the
option "force-set-var". If you know all the variables you will need, this
directive is a good way to register them without letting an agent doing what
it want. This is only required if these variables are not referenced anywhere
in the HAProxy configuration or the SPOE one.
Arguments:
<var name> is a variable name without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
The prefix will be automatically added during the registration. You can have
many "register-var-names" lines.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
MINOR: spoe: add register-var-names directive in spoe-agent configuration In addition to "option force-set-var", recently added, this directive can be used to selectivelly register unknown variable names, without totally relaxing their registration during the runtime, like "option force-set-var" does. So there is no way for a malicious agent to exhaust memory by defining a too high number of variable names. In other hand, you need to enumerate all variable names. This could be painfull in some circumstances. Remember, this directive is only usefull when the variable names are not referenced anywhere in the HAProxy configuration or the SPOE one. Thanks to Etienne Carrire for his help on this part.
2017-12-22 09:00:55 +00:00
See also: "option force-set-var", "option var-prefix".
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
timeout hello <timeout>
Set the maximum time to wait for an agent to receive the AGENT-HELLO frame.
MINOR: spoe: Add 'timeout processing' option to limit time to process an event It is a way to set the maximum time to wait for a stream to process an event, i.e to acquire a stream to talk with an agent, to encode all messages, to send the NOTIFY frame, to receive the corrsponding acknowledgement and to process all actions. It is applied on the stream that handle the client and the server sessions.
2016-11-10 14:04:51 +00:00
It is applied on the stream that handle the connection with the agent.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
Arguments :
<timeout> is the timeout value specified in milliseconds by default, but
can be in any other unit if the number is suffixed by the unit,
as explained at the top of this document.
This timeout is an applicative timeout. It differ from "timeout connect"
defined on backends.
timeout idle <timeout>
MINOR: spoe: Add 'timeout processing' option to limit time to process an event It is a way to set the maximum time to wait for a stream to process an event, i.e to acquire a stream to talk with an agent, to encode all messages, to send the NOTIFY frame, to receive the corrsponding acknowledgement and to process all actions. It is applied on the stream that handle the client and the server sessions.
2016-11-10 14:04:51 +00:00
Set the maximum time to wait for an agent to close an idle connection. It is
applied on the stream that handle the connection with the agent.
Arguments :
<timeout> is the timeout value specified in milliseconds by default, but
can be in any other unit if the number is suffixed by the unit,
as explained at the top of this document.
timeout processing <timeout>
Set the maximum time to wait for a stream to process an event, i.e to acquire
a stream to talk with an agent, to encode all messages, to send the NOTIFY
frame, to receive the corrsponding acknowledgement and to process all
actions. It is applied on the stream that handle the client and the server
sessions.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
Arguments :
<timeout> is the timeout value specified in milliseconds by default, but
can be in any other unit if the number is suffixed by the unit,
as explained at the top of this document.
use-backend <backend>
Specify the backend to use. It must be defined.
Arguments :
<backend> is the name of a valid "backend" section.
2.3. "spoe-message" section
----------------------------
To offload the stream processing, SPOE will send messages with specific
information at a specific moment in the stream life and will wait for
corresponding replies to know what to do.
spoe-message <name>
Create a new SPOE message with the name <name>.
Arguments :
<name> is the name of the SPOE message.
Here you define a message that can be referenced in a "spoe-agent"
section. Following keywords are supported :
MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections.
2017-09-04 13:41:09 +00:00
- acl
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
- args
- event
See also: "spoe-agent" section.
MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections.
2017-09-04 13:41:09 +00:00
acl <aclname> <criterion> [flags] [operator] <value> ...
Declare or complete an access list.
See section 7 about ACL usage in the HAProxy Configuration Manual.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
args [name=]<sample> ...
Define arguments passed into the SPOE message.
Arguments :
<sample> is a sample expression.
When the message is processed, if a sample expression is not available, it is
set to NULL. Arguments are processed in their declaration order and added in
DOC: fix a few typos in the documentation This commit deals with a few misspells in the documentation.
2018-11-14 00:55:16 +00:00
the message in that order. It is possible to declare named arguments.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
For example:
args frontend=fe_id src dst
MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections.
2017-09-04 13:41:09 +00:00
event <name> [ { if | unless } <condition> ]
Set the event that triggers sending of the message. It may optionally be
followed by an ACL-based condition, in which case it will only be evaluated
if the condition is true.
ACL-based conditions are executed in the context of the stream that handle
the client and the server connections.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections.
2017-09-04 13:41:09 +00:00
Arguments :
<name> is the event name.
<condition> is a standard ACL-based condition.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
Supported events are:
- on-client-session
DOC: Fix some typo in SPOE documentation [wt: backport to 1.7]
2016-12-09 16:41:54 +00:00
- on-server-session
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
- on-frontend-tcp-request
- on-backend-tcp-request
- on-tcp-response
- on-frontend-http-request
- on-backend-http-request
- on-http-response
MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections.
2017-09-04 13:41:09 +00:00
See section "Events & Messages" for more details about supported events.
See section 7 about ACL usage in the HAProxy Configuration Manual.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file For now, this section is only parsed. It should have the following format: spoe-group <grp-name> messages <msg-name> ... And then SPOE groups must be referenced in spoe-agent section: spoe-agnt <name> ... groups <grp-name> ... The purpose of these groups is to trigger messages sending from TCP or HTTP rules, directly from HAProxy configuration, and not on specific event. This part will be added in another patch. It is important to note that a message belongs at most to a group.
2017-09-21 08:23:10 +00:00
2.4. "spoe-group" section
--------------------------
This section can be used to declare a group of SPOE messages. Unlike messages
referenced in a "spoe-agent" section, messages inside a group are not sent on a
specific event. The sending must be triggered by TCP or HTTP rules, from the
HAProxy configuration.
spoe-group <name>
Create a new SPOE group with the name <name>.
Arguments :
<name> is the name of the SPOE group.
Here you define a group of SPOE messages that can be referenced in a
"spoe-agent" section. Following keywords are supported :
- messages
See also: "spoe-agent" and "spoe-message" sections.
messages <msg-name> ...
Declare the list of SPOE messages belonging to the group.
Arguments :
<msg-name> is the name of a SPOE message.
Messages declared here must be found in the same engine scope, else an error
is triggered during the configuration parsing. Furthermore, a message belongs
at most to a group. You can have many "messages" lines.
See also: "spoe-message" section.
2.5. Example
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
-------------
Here is a simple but complete example that sends client-ip address to a ip
reputation service. This service can set the variable "ip_score" which is an
integer between 0 and 100, indicating its reputation (100 means totally safe
and 0 a blacklisted IP with no doubt).
###
### HAProxy configuration
frontend www
mode http
bind *:80
filter spoe engine ip-reputation config spoe-ip-reputation.conf
# Reject connection if the IP reputation is under 20
tcp-request content reject if { var(sess.iprep.ip_score) -m int lt 20 }
default_backend http-servers
backend http-servers
mode http
server http A.B.C.D:80
backend iprep-servers
mode tcp
balance roundrobin
timeout connect 5s # greater than hello timeout
timeout server 3m # greater than idle timeout
server iprep1 A1.B1.C1.D1:12345
server iprep2 A2.B2.C2.D2:12345
####
### spoe-ip-reputation.conf
[ip-reputation]
spoe-agent iprep-agent
messages get-ip-reputation
option var-prefix iprep
MINOR: spoe: Remove useless 'timeout ack' option To limit the time to process an event, you should set 'timeout processing' option. So 'timeout ack' option is redundant and useless.
2016-11-19 15:47:56 +00:00
timeout hello 2s
timeout idle 2m
timeout processing 10ms
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
use-backend iprep-servers
spoe-message get-ip-reputation
args ip=src
MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages Now, it is possible to conditionnaly send a SPOE message by adding an ACL-based condition on the "event" line, in a "spoe-message" section. Here is the example coming for the SPOE documentation: spoe-message get-ip-reputation args ip=src event on-client-session if ! { src -f /etc/haproxy/whitelist.lst } To avoid mixin with proxy's ACLs, each SPOE message has its private ACL list. It possible to declare named ACLs in "spoe-message" section, using the same syntax than for proxies. So we can rewrite the previous example to use a named ACL: spoe-message get-ip-reputation args ip=src acl ip-whitelisted src -f /etc/haproxy/whitelist.lst event on-client-session if ! ip-whitelisted ACL-based conditions are executed in the context of the stream that handle the client and the server connections.
2017-09-04 13:41:09 +00:00
event on-client-session if ! { src -f /etc/haproxy/whitelist.lst }
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
3. SPOP specification
----------------------
3.1. Data types
----------------
Here is the bytewise representation of typed data:
TYPED-DATA : <TYPE:4 bits><FLAGS:4 bits><DATA>
Supported types and their representation are:
TYPE | ID | DESCRIPTION
-----------------------------+-----+----------------------------------
NULL | 0 | NULL : <0>
Boolean | 1 | BOOL : <1+FLAG>
32bits signed integer | 2 | INT32 : <2><VALUE:varint>
32bits unsigned integer | 3 | UINT32 : <3><VALUE:varint>
64bits signed integer | 4 | INT64 : <4><VALUE:varint>
32bits unsigned integer | 5 | UNIT64 : <5><VALUE:varint>
IPV4 | 6 | IPV4 : <6><STRUCT IN_ADDR:4 bytes>
IPV6 | 7 | IPV6 : <7><STRUCT IN_ADDR6:16 bytes>
String | 8 | STRING : <8><LENGTH:varint><BYTES>
Binary | 9 | BINARY : <9><LENGTH:varint><BYTES>
10 -> 15 unused/reserved | - | -
-----------------------------+-----+----------------------------------
Variable-length integer (varint) are encoded using Peers encoding:
0 <= X < 240 : 1 byte (7.875 bits) [ XXXX XXXX ]
240 <= X < 2288 : 2 bytes (11 bits) [ 1111 XXXX ] [ 0XXX XXXX ]
2288 <= X < 264432 : 3 bytes (18 bits) [ 1111 XXXX ] [ 1XXX XXXX ] [ 0XXX XXXX ]
264432 <= X < 33818864 : 4 bytes (25 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*2 [ 0XXX XXXX ]
33818864 <= X < 4328786160 : 5 bytes (32 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*3 [ 0XXX XXXX ]
...
For booleans, the value (true or false) is the first bit in the FLAGS
bitfield. if this bit is set to 0, then the boolean is evaluated as false,
otherwise, the boolean is evaluated as true.
3.2. Frames
------------
Exchange between HAProxy and agents are made using FRAME packets. All frames
must be prefixed with their size encoded on 4 bytes in network byte order:
<FRAME-LENGTH:4 bytes> <FRAME>
A frame always starts with its type, on one byte, followed by metadata
containing flags, on 4 bytes and a two variable-length integer representing the
stream identifier and the frame identifier inside the stream:
FRAME : <FRAME-TYPE:1 byte> <METADATA> <FRAME-PAYLOAD>
METADATA : <FLAGS:4 bytes> <STREAM-ID:varint> <FRAME-ID:varint>
Then comes the frame payload. Depending on the frame type, the payload can be
of three types: a simple key/value list, a list of messages or a list of
actions.
FRAME-PAYLOAD : <LIST-OF-MESSAGES> | <LIST-OF-ACTIONS> | <KV-LIST>
LIST-OF-MESSAGES : [ <MESSAGE-NAME> <NB-ARGS:1 byte> <KV-LIST> ... ]
MESSAGE-NAME : <STRING>
LIST-OF-ACTIONS : [ <ACTION-TYPE:1 byte> <NB-ARGS:1 byte> <ACTION-ARGS> ... ]
ACTION-ARGS : [ <TYPED-DATA>... ]
KV-LIST : [ <KV-NAME> <KV-VALUE> ... ]
KV-NAME : <STRING>
KV-VALUE : <TYPED-DATA>
BUG/MEDIUM: spoe: Flags are not encoded in network order The flags are direct copy of the "unsigned int" in the network stream, so the stream contains a 32 bits field encoded with the host endian. - This is not reliable for stream betwen different architecture host - For x86, the bits doesn't correspond to the documentation. This patch add some precision in the documentation and put the bitfield in the stream usig network butes order. Warning: this patch can break compatibility with existing agents. This patch should be backported in all version supporing SPOE Original network capture: 12:28:16.181343 IP 127.0.0.1.46782 > 127.0.0.1.12345: Flags [P.], seq 134:168, ack 59, win 342, options [nop,nop,TS val 2855241281 ecr 2855241281], length 34 0x0000: 4500 0056 6b94 4000 4006 d10b 7f00 0001 E..Vk.@.@....... 0x0010: 7f00 0001 b6be 3039 a3d1 ee54 7d61 d6f7 ......09...T}a.. 0x0020: 8018 0156 fe4a 0000 0101 080a aa2f 8641 ...V.J......./.A 0x0030: aa2f 8641 0000 001e 0301 0000 0000 010f ./.A............ ^^^^^^^^^^ 0x0040: 6368 6563 6b2d 636c 6965 6e74 2d69 7001 check-client-ip. 0x0050: 0006 7f00 0001 ...... Fixed network capture: 12:24:26.948165 IP 127.0.0.1.46706 > 127.0.0.1.12345: Flags [P.], seq 4066280627:4066280661, ack 3148908096, win 342, options [nop,nop,TS val 2855183972 ecr 2855177690], length 34 0x0000: 4500 0056 0538 4000 4006 3768 7f00 0001 E..V.8@.@.7h.... 0x0010: 7f00 0001 b672 3039 f25e 84b3 bbb0 8640 .....r09.^.....@ 0x0020: 8018 0156 fe4a 0000 0101 080a aa2e a664 ...V.J.........d 0x0030: aa2e 8dda 0000 001e 0300 0000 0114 010f ................ ^^^^^^^^^^ 0x0040: 6368 6563 6b2d 636c 6965 6e74 2d69 7001 check-client-ip. 0x0050: 0006 7f00 0001 ......
2018-05-18 10:25:39 +00:00
FLAGS :
Flags are a 32 bits field. They are encoded on 4 bytes in network byte
order, where the bit 0 is the LSB.
0 1 2-31
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
+---+---+----------+
| | A | |
| F | B | |
| I | O | RESERVED |
| N | R | |
| | T | |
+---+---+----------+
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
FIN: Indicates that this is the final payload fragment. The first fragment
may also be the final fragment.
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
ABORT: Indicates that the processing of the current frame must be
cancelled. This bit should be set on frames with a fragmented
payload. It can be ignore for frames with an unfragemnted
payload. When it is set, the FIN bit must also be set.
DOC: fix a few typos in the documentation This commit deals with a few misspells in the documentation.
2018-11-14 00:55:16 +00:00
Frames cannot exceed a maximum size negotiated between HAProxy and agents
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
during the HELLO handshake. Most of time, payload will be small enough to send
it in one frame. But when supported by the peer, it will be possible to
fragment huge payload on many frames. This ability is announced during the
HELLO handshake and it can be asynmetric (supported by agents but not by
HAProxy or the opposite). The following rules apply to fragmentation:
* An unfragemnted payload consists of a single frame with the FIN bit set.
* A fragemented payload consists of several frames with the FIN bit clear and
terminated by a single frame with the FIN bit set. All these frames must
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
share the same STREAM-ID and FRAME-ID. The first frame must set the right
FRAME-TYPE (e.g, NOTIFY). The following frames must have an unset type (0).
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
Beside the support of fragmented payload by a peer, some payload must not be
fragmented. See below for details.
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
IMPORTANT : The maximum size supported by peers for a frame must be greater
than or equal to 256 bytes.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
3.2.1. Frame capabilities
--------------------------
Here are the list of official capabilities that HAProxy and agents can support:
MAJOR: spoe: Add support of pipelined and asynchronous exchanges with agents Now, HAProxy and agents can announce the support for "pipelining" and/or "async" capabilities during the HELLO handshake. For now, HAProxy always announces the support of both. In addition, in its HELLO frames. HAproxy adds the "engine-id" key. It is a uniq string that identify a SPOE engine. The "pipelining" capability is the ability for a peer to decouple NOTIFY and ACK frames. This is a symmectical capability. To be used, it must be supported by HAproxy and agents. Unlike HTTP pipelining, the ACK frames can be send in any order, but always on the same TCP connection used for the corresponding NOTIFY frame. The "async" capability is similar to the pipelining, but here any TCP connection established between HAProxy and the agent can be used to send ACK frames. if an agent accepts connections from multiple HAProxy, it can use the "engine-id" value to group TCP connections.
2016-12-21 07:58:06 +00:00
* fragmentation: This is the ability for a peer to support fragmented
payload in received frames. This is an asymmectical
capability, it only concerns the peer that announces
it. This is the responsibility to the other peer to use it
or not.
* pipelining: This is the ability for a peer to decouple NOTIFY and ACK
frames. This is a symmectical capability. To be used, it must
be supported by HAproxy and agents. Unlike HTTP pipelining, the
ACK frames can be send in any order, but always on the same TCP
connection used for the corresponding NOTIFY frame.
* async: This ability is similar to the pipelining, but here any TCP
connection established between HAProxy and the agent can be used to
send ACK frames. if an agent accepts connections from multiple
HAProxy, it can use the "engine-id" value to group TCP
connections. See details about HAPROXY-HELLO frame.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
Unsupported or unknown capabilities are silently ignored, when possible.
3.2.2. Frame types overview
----------------------------
Here are types of frame supported by SPOE. Frames sent by HAProxy come first,
then frames sent by agents :
TYPE | ID | DESCRIPTION
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
-----------------------------+-----+-------------------------------------
UNSET | 0 | Used for all frames but the first when a
| | payload is fragmented.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
-----------------------------+-----+-------------------------------------
HAPROXY-HELLO | 1 | Sent by HAProxy when it opens a
| | connection on an agent.
| |
HAPROXY-DISCONNECT | 2 | Sent by HAProxy when it want to close
| | the connection or in reply to an
| | AGENT-DISCONNECT frame
| |
NOTIFY | 3 | Sent by HAProxy to pass information
| | to an agent
-----------------------------+-----+-------------------------------------
AGENT-HELLO | 101 | Reply to a HAPROXY-HELLO frame, when
| | the connection is established
| |
AGENT-DISCONNECT | 102 | Sent by an agent just before closing
| | the connection
| |
ACK | 103 | Sent to acknowledge a NOTIFY frame
-----------------------------+-----+-------------------------------------
Unknown frames may be silently skipped.
3.2.3. Workflow
----------------
* Successful HELLO handshake:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
MINOR: spoe/checks: Add support for SPOP health checks A new "option spop-check" statement has been added to enable server health checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters to talk to servers.
2016-11-07 20:07:38 +00:00
| (healthcheck: false) |
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
| --------------------------> |
| |
| AGENT-HELLO |
| <-------------------------- |
| |
MINOR: spoe/checks: Add support for SPOP health checks A new "option spop-check" statement has been added to enable server health checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters to talk to servers.
2016-11-07 20:07:38 +00:00
* Successful HELLO healthcheck:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
| (healthcheck: true) |
| --------------------------> |
| |
| AGENT-HELLO + close() |
| <-------------------------- |
| |
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
* Error encountered by agent during the HELLO handshake:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
| --------------------------> |
| |
| DISCONNECT + close() |
| <-------------------------- |
| |
* Error encountered by HAProxy during the HELLO handshake:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
| --------------------------> |
| |
| AGENT-HELLO |
| <-------------------------- |
| |
| DISCONNECT |
| --------------------------> |
| |
| DISCONNECT + close() |
| <-------------------------- |
| |
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
* Notify / Ack exchange (unfragmented payload):
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
HAPROXY AGENT SRV
| NOTIFY |
| --------------------------> |
| |
| ACK |
| <-------------------------- |
| |
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
* Notify / Ack exchange (fragmented payload):
HAPROXY AGENT SRV
| NOTIFY (frag 1) |
| --------------------------> |
| |
| UNSET (frag 2) |
| --------------------------> |
| ... |
| UNSET (frag N) |
| --------------------------> |
| |
| ACK |
| <-------------------------- |
| |
* Aborted fragmentation of a NOTIFY frame:
HAPROXY AGENT SRV
| ... |
| UNSET (frag X) |
| --------------------------> |
| |
| ACK/ABORT |
| <-------------------------- |
| |
| UNSET (frag X+1) |
| -----------X |
| |
| |
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
* Connection closed by haproxy:
HAPROXY AGENT SRV
| DISCONNECT |
| --------------------------> |
| |
| DISCONNECT + close() |
| <-------------------------- |
| |
* Connection closed by agent:
HAPROXY AGENT SRV
| DISCONNECT + close() |
| <-------------------------- |
| |
3.2.4. Frame: HAPROXY-HELLO
----------------------------
This frame is the first one exchanged between HAProxy and an agent, when the
connection is established. The payload of this frame is a KV-LIST. It cannot be
fragmented. STREAM-ID and FRAME-ID are must be set 0.
Following items are mandatory in the KV-LIST:
* "supported-versions" <STRING>
Last SPOP major versions supported by HAProxy. It is a comma-separated list
of versions, following the format "Major.Minor". Spaces must be ignored, if
any. When a major version is announced by HAProxy, it means it also support
all previous minor versions.
Example: "2.0, 1.5" means HAProxy supports SPOP 2.0 and 1.0 to 1.5
* "max-frame-size" <UINT32>
This is the maximum size allowed for a frame. The HAPROXY-HELLO frame must
be lower or equal to this value.
* "capabilities" <STRING>
This a comma-separated list of capabilities supported by HAProxy. Spaces
must be ignored, if any.
MINOR: spoe/checks: Add support for SPOP health checks A new "option spop-check" statement has been added to enable server health checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters to talk to servers.
2016-11-07 20:07:38 +00:00
Following optional items can be added in the KV-LIST:
* "healthcheck" <BOOLEAN>
If this item is set to TRUE, then the HAPROXY-HELLO frame is sent during a
SPOE health check. When set to FALSE, this item can be ignored.
MAJOR: spoe: Add support of pipelined and asynchronous exchanges with agents Now, HAProxy and agents can announce the support for "pipelining" and/or "async" capabilities during the HELLO handshake. For now, HAProxy always announces the support of both. In addition, in its HELLO frames. HAproxy adds the "engine-id" key. It is a uniq string that identify a SPOE engine. The "pipelining" capability is the ability for a peer to decouple NOTIFY and ACK frames. This is a symmectical capability. To be used, it must be supported by HAproxy and agents. Unlike HTTP pipelining, the ACK frames can be send in any order, but always on the same TCP connection used for the corresponding NOTIFY frame. The "async" capability is similar to the pipelining, but here any TCP connection established between HAProxy and the agent can be used to send ACK frames. if an agent accepts connections from multiple HAProxy, it can use the "engine-id" value to group TCP connections.
2016-12-21 07:58:06 +00:00
* "engine-id" <STRING>
This is a uniq string that identify a SPOE engine.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
To finish the HELLO handshake, the agent must return an AGENT-HELLO frame with
its supported SPOP version, the lower value between its maximum size allowed
for a frame and the HAProxy one and capabilities it supports. If an error
occurs or if an incompatibility is detected with the agent configuration, an
AGENT-DISCONNECT frame must be returned.
3.2.5. Frame: AGENT-HELLO
--------------------------
This frame is sent in reply to a HAPROXY-HELLO frame to finish a HELLO
handshake. As for HAPROXY-HELLO frame, STREAM-ID and FRAME-ID are also set
0. The payload of this frame is a KV-LIST and it cannot be fragmented.
Following items are mandatory in the KV-LIST:
* "version" <STRING>
This is the SPOP version the agent supports. It must follow the format
"Major.Minor" and it must be lower or equal than one of major versions
announced by HAProxy.
* "max-frame-size" <UINT32>
This is the maximum size allowed for a frame. It must be lower or equal to
the value in the HAPROXY-HELLO frame. This value will be used for all
subsequent frames.
* "capabilities" <STRING>
This a comma-separated list of capabilities supported by agent. Spaces must
be ignored, if any.
At this time, if everything is ok for HAProxy (supported version and valid
max-frame-size value), the HELLO handshake is successfully completed. Else,
HAProxy sends a HAPROXY-DISCONNECT frame with the corresponding error.
MINOR: spoe/checks: Add support for SPOP health checks A new "option spop-check" statement has been added to enable server health checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters to talk to servers.
2016-11-07 20:07:38 +00:00
If "healthcheck" item was set to TRUE in the HAPROXY-HELLO frame, the agent can
safely close the connection without DISCONNECT frame. In all cases, HAProxy
DOC: assorted typo fixes in the documentation this is 10th iteration of typo fixes
2020-06-21 16:18:27 +00:00
will close the connection at the end of the health check.
MINOR: spoe/checks: Add support for SPOP health checks A new "option spop-check" statement has been added to enable server health checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters to talk to servers.
2016-11-07 20:07:38 +00:00
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
3.2.6. Frame: NOTIFY
---------------------
Information are sent to the agents inside NOTIFY frames. These frames are
attached to a stream, so STREAM-ID and FRAME-ID must be set. The payload of
NOTIFY frames is a LIST-OF-MESSAGES and, if supported by agents, it can be
fragmented.
NOTIFY frames must be acknowledge by agents sending an ACK frame, repeating
right STREAM-ID and FRAME-ID.
3.2.7. Frame: ACK
------------------
ACK frames must be sent by agents to reply to NOTIFY frames. STREAM-ID and
FRAME-ID found in a NOTIFY frame must be reuse in the corresponding ACK
frame. The payload of ACK frames is a LIST-OF-ACTIONS and, if supported by
HAProxy, it can be fragmented.
3.2.8. Frame: HAPROXY-DISCONNECT
---------------------------------
If an error occurs, at anytime, from the HAProxy side, a HAPROXY-DISCONNECT
frame is sent with information describing the error. HAProxy will wait an
AGENT-DISCONNECT frame in reply. All other frames will be ignored. The agent
must then close the socket.
The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and
FRAME-ID are must be set 0.
Following items are mandatory in the KV-LIST:
* "status-code" <UINT32>
This is the code corresponding to the error.
* "message" <STRING>
This is a textual message describing the error.
For more information about known errors, see section "Errors & timeouts"
3.2.9. Frame: AGENT-DISCONNECT
-------------------------------
If an error occurs, at anytime, from the agent size, a AGENT-DISCONNECT frame
DOC: fix typos s/accidently/accidentally/ s/any ot these messages/any of theses messages/ s/catched/caught/ s/completly/completely/ s/convertor/converter/ s/desribing/describing/ s/developper/developer/ s/eventhough/even though/ s/exectution/execution/ s/functionnality/functionality/ s/If it receive a/If it receives a/ s/In can even/It can even/ s/informations/information/ s/it will be remove /it will be removed / s/langage/language/ s/mentionned/mentioned/ s/negociated/negotiated/ s/Optionnaly/Optionally/ s/ouputs/outputs/ s/outweights/outweighs/ s/ressources/resources/
2019-05-24 08:25:45 +00:00
is sent, with information describing the error. such frame is also sent in reply
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
to a HAPROXY-DISCONNECT. The agent must close the socket just after sending
this frame.
The payload of this frame is a KV-LIST. It cannot be fragmented. STREAM-ID and
FRAME-ID are must be set 0.
Following items are mandatory in the KV-LIST:
* "status-code" <UINT32>
This is the code corresponding to the error.
* "message" <STRING>
This is a textual message describing the error.
For more information about known errors, see section "Errors & timeouts"
3.3. Events & Messages
-----------------------
Information about streams are sent in NOTIFY frames. You can specify which kind
of information to send by defining "spoe-message" sections in your SPOE
configuration file. for each "spoe-message" there will be a message in a NOTIFY
frame when the right event is triggered.
A NOTIFY frame is sent for an specific event when there is at least one
"spoe-message" attached to this event. All messages for an event will be added
in the same NOTIFY frame.
Here is the list of supported events:
* on-client-session is triggered when a new client session is created.
This event is only available for SPOE filters
declared in a frontend or a listen section.
* on-frontend-tcp-request is triggered just before the evaluation of
"tcp-request content" rules on the frontend side.
This event is only available for SPOE filters
declared in a frontend or a listen section.
* on-backend-tcp-request is triggered just before the evaluation of
"tcp-request content" rules on the backend side.
This event is skipped for SPOE filters declared
in a listen section.
* on-frontend-http-request is triggered just before the evaluation of
"http-request" rules on the frontend side. This
event is only available for SPOE filters declared
in a frontend or a listen section.
* on-backend-http-request is triggered just before the evaluation of
"http-request" rules on the backend side. This
event is skipped for SPOE filters declared in a
listen section.
* on-server-session is triggered when the session with the server is
established.
* on-tcp-response is triggered just before the evaluation of
"tcp-response content" rules.
* on-http-response is triggered just before the evaluation of
"http-response" rules.
The stream processing will loop on these events, when triggered, waiting the
agent reply.
3.4. Actions
-------------
An agent must acknowledge each NOTIFY frame by sending the corresponding ACK
frame. Actions can be added in these frames to dynamically take action on the
processing of a stream.
Here is the list of supported actions:
* set-var set the value for an existing variable. 3 arguments must be
attached to this action: the variable scope (proc, sess, txn,
DOC: SPOE.txt: fix a typo
2018-06-01 03:38:00 +00:00
req or res), the variable name (a string) and its value.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
ACTION-SET-VAR : <SET-VAR:1 byte><NB-ARGS:1 byte><VAR-SCOPE:1 byte><VAR-NAME><VAR-VALUE>
SET-VAR : <1>
NB-ARGS : <3>
VAR-SCOPE : <PROCESS> | <SESSION> | <TRANSACTION> | <REQUEST> | <RESPONSE>
VAR-NAME : <STRING>
VAR-VALUE : <TYPED-DATA>
PROCESS : <0>
SESSION : <1>
TRANSACTION : <2>
REQUEST : <3>
MAJOR: spoe: Add support of pipelined and asynchronous exchanges with agents Now, HAProxy and agents can announce the support for "pipelining" and/or "async" capabilities during the HELLO handshake. For now, HAProxy always announces the support of both. In addition, in its HELLO frames. HAproxy adds the "engine-id" key. It is a uniq string that identify a SPOE engine. The "pipelining" capability is the ability for a peer to decouple NOTIFY and ACK frames. This is a symmectical capability. To be used, it must be supported by HAproxy and agents. Unlike HTTP pipelining, the ACK frames can be send in any order, but always on the same TCP connection used for the corresponding NOTIFY frame. The "async" capability is similar to the pipelining, but here any TCP connection established between HAProxy and the agent can be used to send ACK frames. if an agent accepts connections from multiple HAProxy, it can use the "engine-id" value to group TCP connections.
2016-12-21 07:58:06 +00:00
RESPONSE : <4>
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
* unset-var unset the value for an existing variable. 2 arguments must be
attached to this action: the variable scope (proc, sess, txn,
DOC: SPOE.txt: fix a typo
2018-06-01 03:38:00 +00:00
req or res) and the variable name (a string).
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
DOC: Fix some typo in SPOE documentation [wt: backport to 1.7]
2016-12-09 16:41:54 +00:00
ACTION-UNSET-VAR : <UNSET-VAR:1 byte><NB-ARGS:1 byte><VAR-SCOPE:1 byte><VAR-NAME>
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
DOC: Fix some typo in SPOE documentation [wt: backport to 1.7]
2016-12-09 16:41:54 +00:00
UNSET-VAR : <2>
NB-ARGS : <2>
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
VAR-SCOPE : <PROCESS> | <SESSION> | <TRANSACTION> | <REQUEST> | <RESPONSE>
VAR-NAME : <STRING>
PROCESS : <0>
SESSION : <1>
TRANSACTION : <2>
REQUEST : <3>
MAJOR: spoe: Add support of pipelined and asynchronous exchanges with agents Now, HAProxy and agents can announce the support for "pipelining" and/or "async" capabilities during the HELLO handshake. For now, HAProxy always announces the support of both. In addition, in its HELLO frames. HAproxy adds the "engine-id" key. It is a uniq string that identify a SPOE engine. The "pipelining" capability is the ability for a peer to decouple NOTIFY and ACK frames. This is a symmectical capability. To be used, it must be supported by HAproxy and agents. Unlike HTTP pipelining, the ACK frames can be send in any order, but always on the same TCP connection used for the corresponding NOTIFY frame. The "async" capability is similar to the pipelining, but here any TCP connection established between HAProxy and the agent can be used to send ACK frames. if an agent accepts connections from multiple HAProxy, it can use the "engine-id" value to group TCP connections.
2016-12-21 07:58:06 +00:00
RESPONSE : <4>
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
NOTE: Name of the variables will be automatically prefixed by HAProxy to avoid
name clashes with other variables used in HAProxy. Moreover, unknown
variable will be silently ignored.
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
3.5. Errors & timeouts
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
----------------------
Here is the list of all known errors:
STATUS CODE | DESCRIPTION
----------------+--------------------------------------------------------
0 | normal (no error occurred)
1 | I/O error
2 | A timeout occurred
3 | frame is too big
4 | invalid frame received
5 | version value not found
6 | max-frame-size value not found
7 | capabilities value not found
8 | unsupported version
9 | max-frame-size too big or too small
DOC: spoe: Update SPOE documentation to reflect recent changes
2017-02-27 20:59:39 +00:00
10 | payload fragmentation is not supported
11 | invalid interlaced frames
12 | frame-id not found (it does not match any referenced frame)
13 | resource allocation error
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
99 | an unknown error occurrde
----------------+--------------------------------------------------------
An agent can define its own errors using a not yet assigned status code.
MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section By default, for a specific stream, when an abnormal/unexpected error occurs, the SPOE is disabled for all the transaction. So if you have several events configured, such error on an event will disabled all followings. For TCP streams, this will disable the SPOE for the whole session. For HTTP streams, this will disable it for the transaction (request and response). To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent' section. With this option, only the current event will be ignored.
2016-11-14 09:54:21 +00:00
IMPORTANT NOTE: By default, for a specific stream, when an abnormal/unexpected
error occurs, the SPOE is disabled for all the transaction. So
if you have several events configured, such error on an event
DOC: assorted typo fixes in the documentation this is 10th iteration of typo fixes
2020-06-21 16:18:27 +00:00
will disabled all following. For TCP streams, this will
MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section By default, for a specific stream, when an abnormal/unexpected error occurs, the SPOE is disabled for all the transaction. So if you have several events configured, such error on an event will disabled all followings. For TCP streams, this will disable the SPOE for the whole session. For HTTP streams, this will disable it for the transaction (request and response). To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent' section. With this option, only the current event will be ignored.
2016-11-14 09:54:21 +00:00
disable the SPOE for the whole session. For HTTP streams, this
will disable it for the transaction (request and response).
See 'option continue-on-error' to bypass this limitation.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
DOC: fix a few typos in the documentation This commit deals with a few misspells in the documentation.
2018-11-14 00:55:16 +00:00
To avoid a stream to wait undefinetly, you must carefully choose the
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
acknowledgement timeout. In most of cases, it will be quiet low. But it depends
on the responsivness of your service.
You must also choose idle timeout carefully. Because connection with your
service depends on the backend configuration used by the SPOA, it is important
to use a lower value for idle timeout than the server timeout. Else the
connection will be closed by HAProxy. The same is true for hello timeout. You
should choose a lower value than the connect timeout.
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
4. Logging
-----------
Activity of an SPOE is logged using HAProxy's logger. The messages are logged
in the context of the streams that handle the client and the server
connections. A message is emitted for each event or group handled by an
SPOE. Depending on the status code, the log level will be different. In the
normal case, when no error occurred, the message is logged with the level
LOG_NOTICE. Otherwise, the message is logged with the level LOG_WARNING.
MINOR: spoe: use agent's logger to log SPOE messages Instead of using the logger of the stream, we now use dedicated logger of the SPOE. This means a logger should be defined.
2018-03-26 15:20:58 +00:00
The messages are logged using the agent's logger, if defined, and use the
following format:
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
DOC: spoe: fix a typo s/STATUC/STATUS/
2018-04-26 12:25:43 +00:00
SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUS-CODE reqT/qT/wT/resT/pT \
MINOR: spoe: Add counters to log info about SPOE agents In addition to metrics about time spent in the SPOE, following counters have been added: * applets : number of SPOE applets. * idles : number of idle applets. * nb_sending : number of streams waiting to send data. * nb_waiting : number of streams waiting for a ack. * nb_processed : number of events/groups processed by the SPOE (from the stream point of view). * nb_errors : number of errors during the processing (from the stream point of view). Log messages has been updated to report these counters. Following pattern has been added at the end of the log message: ... <idles>/<applets> <nb_sending>/<nb_waiting> <nb_error>/<nb_processed>
2018-04-04 08:25:50 +00:00
<idles>/<applets> <nb_sending>/<nb_waiting> <nb_error>/<nb_processed>
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
AGENT is the agent name
TYPE is EVENT of GROUP
NAME is the event or the group name
STREAM-ID is an integer, the unique id of the stream
STATUS_CODE is the processing's status code
reqT/qT/wT/resT/pT are the following time events:
* reqT : the encoding time. It includes ACLs processing, if any. For
fragmented frames, it is the sum of all fragments.
* qT : the delay before the request gets out the sending queue. For
fragmented frames, it is the sum of all fragments.
DOC: fix a few typos in the documentation This commit deals with a few misspells in the documentation.
2018-11-14 00:55:16 +00:00
* wT : the delay before the response is received. No fragmentation
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
supported here.
* resT : the delay to process the response. No fragmentation supported
MINOR: spoe: Add options to store processing times in variables "set-process-time" and "set-total-time" options have been added to store processing times in the transaction scope, at each event and group processing, the current one and the total one. So it is possible to get them. TODO: documentation
2018-03-22 08:08:20 +00:00
here.
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
* pT : the delay to process the event or the group. From the stream
MINOR: spoe: Add options to store processing times in variables "set-process-time" and "set-total-time" options have been added to store processing times in the transaction scope, at each event and group processing, the current one and the total one. So it is possible to get them. TODO: documentation
2018-03-22 08:08:20 +00:00
point of view, it is the latency added by the SPOE processing.
It is more or less the sum of values above.
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
MINOR: spoe: Add counters to log info about SPOE agents In addition to metrics about time spent in the SPOE, following counters have been added: * applets : number of SPOE applets. * idles : number of idle applets. * nb_sending : number of streams waiting to send data. * nb_waiting : number of streams waiting for a ack. * nb_processed : number of events/groups processed by the SPOE (from the stream point of view). * nb_errors : number of errors during the processing (from the stream point of view). Log messages has been updated to report these counters. Following pattern has been added at the end of the log message: ... <idles>/<applets> <nb_sending>/<nb_waiting> <nb_error>/<nb_processed>
2018-04-04 08:25:50 +00:00
<idle> is the numbers of idle SPOE applets
<applets> is the numbers of SPOE applets
<nb_sending> is the numbers of streams waiting to send data
<nb_waiting> is the numbers of streams waiting for a ack
<nb_error> is the numbers of processing errors
<nb_processed> is the numbers of events/groups processed
MINOR: spoe: Add metrics in to know time spent in the SPOE Following metrics are added for each event or group of messages processed in the SPOE: * processing time: the delay to process the event or the group. From the stream point of view, it is the latency added by the SPOE processing. * request time : It is the encoding time. It includes ACLs processing, if any. For fragmented frames, it is the sum of all fragments. * queue time : the delay before the request gets out the sending queue. For fragmented frames, it is the sum of all fragments. * waiting time: the delay before the reponse is received. No fragmentation supported here. * response time: the delay to process the response. No fragmentation supported here. * total time: (unused for now). It is the sum of all events or groups processed by the SPOE for a specific threads. Log messages has been updated. Before, only errors was logged (status_code != 0). Now every processing is logged, following this format: SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUC-CODE reqT/qT/wT/resT/pT where: AGENT is the agent name TYPE is EVENT of GROUP NAME is the event or the group name STREAM-ID is an integer, the unique id of the stream STATUS_CODE is the processing's status code reqT/qT/wT/resT/pT are delays descrive above For all these delays, -1 means the processing was interrupted before the end. So -1 for the queue time means the request was never dequeued. For fragmented frames it is harder to know when the interruption happened. For now, messages are logged using the same logger than the backend of the stream which initiated the request.
2018-03-22 08:07:41 +00:00
For all these time events, -1 means the processing was interrupted before the
end. So -1 for the queue time means the request was never dequeued. For
fragmented frames it is harder to know when the interruption happened.
MAJOR: spoe: Add an experimental Stream Processing Offload Engine SPOE makes possible the communication with external components to retrieve some info using an in-house binary protocol, the Stream Processing Offload Protocol (SPOP). In the long term, its aim is to allow any kind of offloading on the streams. This first version, besides being experimental, won't do lot of things. The most important today is to validate the protocol design and lay the foundations of what will, one day, be a full offload engine for the stream processing. So, for now, the SPOE can offload the stream processing before "tcp-request content", "tcp-response content", "http-request" and "http-response" rules. And it only supports variables creation/suppression. But, in spite of these limited features, we can easily imagine to implement a SSO solution, an ip reputation service or an ip geolocation service. Internally, the SPOE is implemented as a filter. So, to use it, you must use following line in a proxy proxy section: frontend my-front ... filter spoe [engine <name>] config <file> ... It uses its own configuration file to keep the HAProxy configuration clean. It is also a easy way to disable it by commenting out the filter line. See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-10-27 20:29:49 +00:00
/*
* Local variables:
* fill-column: 79
* End:
*/