PROMEX: A Prometheus exporter for HAProxy
-------------------------------------------

Prometheus is a monitoring and alerting system. More and more people use it to
monitor their environment (this is written February 2019). It collects metrics
from monitored targets by scraping metrics HTTP endpoints on these targets. For
HAProxy, The Prometheus team officially supports an exporter written in Go
(https://github.com/prometheus/haproxy_exporter). But it requires an extra
software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus
exporter for HAProxy. It was developed as a service and is directly available in
HAProxy, like the stats applet.

However, PROMEX is not built by default with HAProxy. It is provided as an extra
component for everyone want to use it. So you need to explicitly build HAProxy
with the PROMEX service, setting the Makefile variable "USE_PROMEX" to "1". For
instance:

    > make TARGET=linux-glibc USE_PROMEX=1

if HAProxy provides the PROMEX service, the following build option will be
reported by the command "haproxy -vv":

    Built with the Prometheus exporter as a service

To be used, it must be enabled in the configuration with an "http-request" rule
and the corresponding HTTP proxy must enable the HTX support. For instance:

    frontend test
        mode http
        ...
        http-request use-service prometheus-exporter if { path /metrics }
        ...


This service has been developed as a third-party component because it could
become obsolete, depending on how much time Prometheus will remain heavily
used. This is said with no ulterior motive of course. Prometheus is a great
software and I hope all the well for it. But we involve in a environment moving
quickly and a solution may be obvious today could be deprecated the next
year. And because PROMEX is not integrated by default into the HAProxy codebase,
it will need some interest to be actively supported. All contribution of any
kind are welcome.

You must also be careful if you use with huge configurations. Unlike the stats
applet, all metrics are not grouped by service (proxy, listener or server). With
PROMEX, all lines for a given metric are provided as one single group. So
instead of collecting all metrics for a proxy before moving to the next one, we
must loop on all proxies for each metric. Same for the servers. Thus, it will
spend much more resources to produce the Prometheus metrics than the CSV export
through the stats page. To give a comparison order, quick benchmarks shown that
a PROMEX dump is 5x slower and 20x more verbose than a CSV export.


metrics filtering
-------------------

It is possible to dynamically select the metrics to export if you don't use all
of them passing parameters in the query-string.

* Filtering on scopes

The metrics may be filtered by scopes. Multiple parameters with "scope" as name
may be passed in the query-string to filter exported metrics, with one of those
values: global, frontend, backend, server or '*' (means all). A scope parameter
with no value means to filter out all scopes (nothing is returned). The scope
parameters are parsed in their appearance order in the query-string. So an empty
scope will reset all scopes already parsed. But it can be overridden by
following scope parameters in the query-string. By default everything is
exported. Here are examples:

  /metrics?scope=server                 # ==> server metrics will be exported
  /metrics?scope=frontend&scope=backend # ==> Frontend and backend metrics will be exported
  /metrics?scope=listener               # ==> listener metrics will be exported
  /metrics?scope=*&scope=               # ==> no metrics will be exported
  /metrics?scope=&scope=global          # ==> global metrics will be exported
  /metrics?scope=sticktable             # ==> stick tables metrics will be exported

* Filtering on metrics name

It is possible to filter metrics dumped by the exporter. To to so, multiple
"metrics" parameters may be passed to specify all metrics to include or exclude,
as a comma-separated list of filter. By default, there is no filter and all
metrics are dumped. By specifying at least one metric to be included in the
dump, this disables the default behavior and only explicitly mentioned metrics
are dumped. To include a metric, its name must be specified. To exclude it, its
name must be preceded by a minus character ('-'). Here are examples:

  # Dumped all metrics, except "haproxy_server_check_status"
  /metrics?metrics=-haproxy_server_check_status

  # Only dump frontends, backends and servers status
  /metrics?metrics=haproxy_frontend_status,haproxy_backend_status,haproxy_server_status

* Add section description as label for all metrics

It is possible to set a description in global and proxy sections, via the
"description" directive. The global descrption is exposed if it is define via
the "haproxy_process_description" metric. But the descriptions provided in proxy
sections are not dumped. However, it is possible to add it as a label for all
metrics of the corresponding section, including the global one. To do so,
"desc-labels" parameter must be set:

  /metrics?desc-labels

  / metrics?scope=frontend&desc-labels

* Dump extra counters

Internally, some modules can register to frontends, backends, servers or
listeners to export extra counters. For instance, some multiplexers do so on
frontends or backends. To display extra counters for all regiestered modules,
"extra-counters" parameter must be pass. It can be cumulated with "scope"
parameters:

  /metrics?extra-counters          # ==> export all extra counters in
                                   #     addition to main ones, for all
                                   #     scopes
  /metrics?scope=frontend&extra-counters # ==> export extra counters for
                                         #     frontends

There are extra counters are only for frontends, backends, servers and
listeners.

* How do I prevent my prometheus instance to explode?

** Filtering on servers state

It is possible to exclude from returned metrics all servers in maintenance mode
passing the parameter "no-maint" in the query-string. This parameter may help to
solve performance issues of configuration that use the server templates to
manage dynamic provisionning. Note there is no consistency check on the servers
state. So, if the state of a server changes while the exporter is running, only
a part of the metrics for this server will be dumped.

prometheus example config:

For server-template users:
- <job>
  params:
    no-maint:
    - empty

** Scrap server health checks only

All health checks status are dump through `state` label values. If you want to
scrap server health check status but prevent all server metrics to be saved,
except the server_check_status, you may configure prometheus that way:

- <job>
   metric_relabel_configs:
   - source_labels: ['__name__']
      regex: 'haproxy_(process_|frontend_|listener_|backend_|server_check_status).*'
      action: keep

Exported metrics
------------------

See prometheus export for the description of each field. Only main metrics are
listed below. Metrics from extra counters are not listed.

* Globals metrics

+------------------------------------------------+
|    Metric name                                 |
+------------------------------------------------+
| haproxy_process_nbthread                       |
| haproxy_process_nbproc                         |
| haproxy_process_relative_process_id            |
| haproxy_process_uptime_seconds                 |
| haproxy_process_pool_failures_total            |
| haproxy_process_max_fds                        |
| haproxy_process_max_sockets                    |
| haproxy_process_max_connections                |
| haproxy_process_hard_max_connections           |
| haproxy_process_current_connections            |
| haproxy_process_connections_total              |
| haproxy_process_requests_total                 |
| haproxy_process_max_ssl_connections            |
| haproxy_process_current_ssl_connections        |
| haproxy_process_ssl_connections_total          |
| haproxy_process_max_pipes                      |
| haproxy_process_pipes_used_total               |
| haproxy_process_pipes_free_total               |
| haproxy_process_current_connection_rate        |
| haproxy_process_limit_connection_rate          |
| haproxy_process_max_connection_rate            |
| haproxy_process_current_session_rate           |
| haproxy_process_limit_session_rate             |
| haproxy_process_max_session_rate               |
| haproxy_process_current_ssl_rate               |
| haproxy_process_limit_ssl_rate                 |
| haproxy_process_max_ssl_rate                   |
| haproxy_process_current_frontend_ssl_key_rate  |
| haproxy_process_max_frontend_ssl_key_rate      |
| haproxy_process_frontend_ssl_reuse             |
| haproxy_process_current_backend_ssl_key_rate   |
| haproxy_process_max_backend_ssl_key_rate       |
| haproxy_process_ssl_cache_lookups_total        |
| haproxy_process_ssl_cache_misses_total         |
| haproxy_process_http_comp_bytes_in_total       |
| haproxy_process_http_comp_bytes_out_total      |
| haproxy_process_limit_http_comp                |
| haproxy_process_current_zlib_memory            |
| haproxy_process_max_zlib_memory                |
| haproxy_process_current_tasks                  |
| haproxy_process_current_run_queue              |
| haproxy_process_idle_time_percent              |
| haproxy_process_node                           |
| haproxy_process_description                    |
| haproxy_process_stopping                       |
| haproxy_process_jobs                           |
| haproxy_process_unstoppable_jobs               |
| haproxy_process_listeners                      |
| haproxy_process_active_peers                   |
| haproxy_process_connected_peers                |
| haproxy_process_dropped_logs_total             |
| haproxy_process_busy_polling_enabled           |
| haproxy_process_failed_resolutions             |
| haproxy_process_bytes_out_total                |
| haproxy_process_spliced_bytes_out_total        |
| haproxy_process_bytes_out_rate                 |
| haproxy_process_recv_logs_total                |
| haproxy_process_build_info                     |
| haproxy_process_max_memory_bytes               |
| haproxy_process_pool_allocated_bytes           |
| haproxy_process_pool_used_bytes                |
| haproxy_process_start_time_seconds             |
+------------------------------------------------+

* Frontend metrics

+-------------------------------------------------+
|    Metric name                                  |
+-------------------------------------------------+
| haproxy_frontend_current_sessions               |
| haproxy_frontend_max_sessions                   |
| haproxy_frontend_limit_sessions                 |
| haproxy_frontend_sessions_total                 |
| haproxy_frontend_bytes_in_total                 |
| haproxy_frontend_bytes_out_total                |
| haproxy_frontend_requests_denied_total          |
| haproxy_frontend_responses_denied_total         |
| haproxy_frontend_request_errors_total           |
| haproxy_frontend_status                         |
| haproxy_frontend_limit_session_rate             |
| haproxy_frontend_max_session_rate               |
| haproxy_frontend_http_responses_total           |
| haproxy_frontend_http_requests_rate_max         |
| haproxy_frontend_http_requests_total            |
| haproxy_frontend_http_comp_bytes_in_total       |
| haproxy_frontend_http_comp_bytes_out_total      |
| haproxy_frontend_http_comp_bytes_bypassed_total |
| haproxy_frontend_http_comp_responses_total      |
| haproxy_frontend_connections_rate_max           |
| haproxy_frontend_connections_total              |
| haproxy_frontend_intercepted_requests_total     |
| haproxy_frontend_denied_connections_total       |
| haproxy_frontend_denied_sessions_total          |
| haproxy_frontend_failed_header_rewriting_total  |
| haproxy_frontend_http_cache_lookups_total       |
| haproxy_frontend_http_cache_hits_total          |
| haproxy_frontend_internal_errors_total          |
+-------------------------------------------------+

* Listener metrics

+-------------------------------------------------+
|    Metric name                                  |
+-------------------------------------------------+
| haproxy_listener_current_sessions               |
| haproxy_listener_max_sessions                   |
| haproxy_listener_limit_sessions                 |
| haproxy_listener_sessions_total                 |
| haproxy_listener_bytes_in_total                 |
| haproxy_listener_bytes_out_total                |
| haproxy_listener_requests_denied_total          |
| haproxy_listener_responses_denied_total         |
| haproxy_listener_request_errors_total           |
| haproxy_listener_status                         |
| haproxy_listener_denied_connections_total       |
| haproxy_listener_denied_sessions_total          |
| haproxy_listener_failed_header_rewriting_total  |
| haproxy_listener_internal_errors_total          |
+-------------------------------------------------+

* Backend metrics

+-----------------------------------------------------+
|    Metric name                                      |
+-----------------------------------------------------+
| haproxy_backend_current_queue                       |
| haproxy_backend_max_queue                           |
| haproxy_backend_current_sessions                    |
| haproxy_backend_max_sessions                        |
| haproxy_backend_limit_sessions                      |
| haproxy_backend_sessions_total                      |
| haproxy_backend_bytes_in_total                      |
| haproxy_backend_bytes_out_total                     |
| haproxy_backend_requests_denied_total               |
| haproxy_backend_responses_denied_total              |
| haproxy_backend_connection_errors_total             |
| haproxy_backend_response_errors_total               |
| haproxy_backend_retry_warnings_total                |
| haproxy_backend_redispatch_warnings_total           |
| haproxy_backend_status                              |
| haproxy_backend_weight                              |
| haproxy_backend_active_servers                      |
| haproxy_backend_backup_servers                      |
| haproxy_backend_check_up_down_total                 |
| haproxy_backend_check_last_change_seconds           |
| haproxy_backend_downtime_seconds_total              |
| haproxy_backend_loadbalanced_total                  |
| haproxy_backend_max_session_rate                    |
| haproxy_backend_http_responses_total                |
| haproxy_backend_http_requests_total                 |
| haproxy_backend_client_aborts_total                 |
| haproxy_backend_server_aborts_total                 |
| haproxy_backend_http_comp_bytes_in_total            |
| haproxy_backend_http_comp_bytes_out_total           |
| haproxy_backend_http_comp_bytes_bypassed_total      |
| haproxy_backend_http_comp_responses_total           |
| haproxy_backend_last_session_seconds                |
| haproxy_backend_queue_time_average_seconds          |
| haproxy_backend_connect_time_average_seconds        |
| haproxy_backend_response_time_average_seconds       |
| haproxy_backend_total_time_average_seconds          |
| haproxy_backend_failed_header_rewriting_total       |
| haproxy_backend_connection_attempts_total           |
| haproxy_backend_connection_reuses_total             |
| haproxy_backend_http_cache_lookups_total            |
| haproxy_backend_http_cache_hits_total               |
| haproxy_backend_max_queue_time_seconds              |
| haproxy_backend_max_connect_time_seconds            |
| haproxy_backend_max_response_time_seconds           |
| haproxy_backend_max_total_time_seconds              |
| haproxy_backend_internal_errors_total               |
| haproxy_backend_uweight                             |
| haproxy_backend_agg_server_status                   |
| haproxy_backend_agg_check_status                    |
+-----------------------------------------------------+

* Server metrics

+----------------------------------------------------+
|    Metric name                                     |
+----------------------------------------------------+
| haproxy_server_current_queue                       |
| haproxy_server_max_queue                           |
| haproxy_server_current_sessions                    |
| haproxy_server_max_sessions                        |
| haproxy_server_limit_sessions                      |
| haproxy_server_sessions_total                      |
| haproxy_server_bytes_in_total                      |
| haproxy_server_bytes_out_total                     |
| haproxy_server_responses_denied_total              |
| haproxy_server_connection_errors_total             |
| haproxy_server_response_errors_total               |
| haproxy_server_retry_warnings_total                |
| haproxy_server_redispatch_warnings_total           |
| haproxy_server_status                              |
| haproxy_server_weight                              |
| haproxy_server_active                              |
| haproxy_server_backup                              |
| haproxy_server_check_failures_total                |
| haproxy_server_check_up_down_total                 |
| haproxy_server_check_last_change_seconds           |
| haproxy_server_downtime_seconds_total              |
| haproxy_server_queue_limit                         |
| haproxy_server_current_throttle                    |
| haproxy_server_loadbalanced_total                  |
| haproxy_server_max_session_rate                    |
| haproxy_server_check_status                        |
| haproxy_server_check_code                          |
| haproxy_server_check_duration_seconds              |
| haproxy_server_http_responses_total                |
| haproxy_server_client_aborts_total                 |
| haproxy_server_server_aborts_total                 |
| haproxy_server_last_session_seconds                |
| haproxy_server_queue_time_average_seconds          |
| haproxy_server_connect_time_average_seconds        |
| haproxy_server_response_time_average_seconds       |
| haproxy_server_total_time_average_seconds          |
| haproxy_server_failed_header_rewriting_total       |
| haproxy_server_connection_attempts_total           |
| haproxy_server_connection_reuses_total             |
| haproxy_server_idle_connections_current            |
| haproxy_server_idle_connections_limit              |
| haproxy_server_max_queue_time_seconds              |
| haproxy_server_max_connect_time_seconds            |
| haproxy_server_max_response_time_seconds           |
| haproxy_server_max_total_time_seconds              |
| haproxy_server_internal_errors_total               |
| haproxy_server_unsafe_idle_connections_current     |
| haproxy_server_safe_idle_connections_current       |
| haproxy_server_used_connections_current            |
| haproxy_server_need_connections_current            |
| haproxy_server_uweight                             |
+----------------------------------------------------+

* Stick table metrics

+----------------------------------------------------+
|    Metric name                                     |
+----------------------------------------------------+
| haproxy_sticktable_size                            |
| haproxy_sticktable_used                            |
+----------------------------------------------------+

* Resolvers metrics

+----------------------------------------------------+
|    Metric name                                     |
+----------------------------------------------------+
| haproxy_resolver_sent                              |
| haproxy_resolver_send_error                        |
| haproxy_resolver_valid                             |
| haproxy_resolver_update                            |
| haproxy_resolver_cname                             |
| haproxy_resolver_cname_error                       |
| haproxy_resolver_any_err                           |
| haproxy_resolver_nx                                |
| haproxy_resolver_timeout                           |
| haproxy_resolver_refused                           |
| haproxy_resolver_other                             |
| haproxy_resolver_invalid                           |
| haproxy_resolver_too_big                           |
| haproxy_resolver_outdated                          |
+----------------------------------------------------+