* Update to Go 1.18.
* Update minimum Go version to 1.17.
* Update Go modules for 1.17 format.
* Bump Go modules
* Enable dependabot.
* Update Prometheus common files.
* Fixup yamllint.
Signed-off-by: SuperQ <superq@gmail.com>
This moves the metrics that are queried from pg_stat_bgwriter into a dedicated collector instead of dynamically generating queries and metrics from a map. It renames some metrics including adding the `_total` suffix on all of the counters to match prometheus standards. This implementation uses QueryRowContext to allow for later addition of context for cancellation. From the Postgres documentation, it states that there is one line per WAL sender process, but it is unclear how to differentiate between them in any meaningful way. When querying the table, there is no column to identify the row, only metrics about bgwriter.
Signed-off-by: Joe Adams <github@joeadams.io>
Converts the pg_database metrics from queries.yaml to a built in collector. This is enabled by default because it is not likely to be a performance problem and likely very useful data.
Signed-off-by: Joe Adams <github@joeadams.io>
Moves code into more manageable, logical files to group behavior together. This should help improve a developer's ability to navigate the code.
Signed-off-by: Joe Adams <github@joeadams.io>
* Run the query for specific database version if provided from yml file.
By default query will run on all the databases if "runonserver" is not provided.
If user want the query to be run on multiple database versions, use below string.
runonserver: "9.5, 9.6"
Example yml file as below. ( e.g. below query will run only on database version 9.5 )
pg_replication:
query: "SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) as lag"
master: true
runonserver: "9.5"
metrics:
- lag:
usage: "GAUGE"
description: "Replication lag behind master in seconds"
* Fixed the below review comments given by Ashesh Vashi
Instead of having db version string from yml file, user can define the range of
database server version where query is to be executed.
If user want to run the query on database version greater than 10.0.0, use below format.
runonserver: ">=10.0.0"
Below are the example of db version range user can defined in yml file.
<=10.1.0
>=12.1.0
=11.0.0
<9.6.0 || >=11.0.0
* Remove the call from unused places where 'runOnServer' is not required.
Only Server type hold that value.
* Fix compilation issues.
* Fix the issue with Debugln to print the database server version
* Support connstring syntax when discovering databases
Support connstring DSNs (`host=... user=... password=... dbname=...`) in
addition to URIs (`postgresql://user:pass@host/dbname`) for purposes of
database discovery.
Connstring syntax is needed to support accessing PostgreSQL via Unix
domain sockets (`host=/run/postgres`), which is not really possible with
URI syntax.
* Appease gometalinter, don't shadow namespace
When the connection to the PostgreSQL instance cannot be established straight
at startup, a race condition can happen when autoDiscoverDatabases is true. If
discoverDatabaseDSNs fails, no dsn is set as the master database, and, if
scrapeDSN succeeds, checkMapVersions will have omitted the default metrics in
the server metric map. The metric map won't be updated unless the version
returned by the PostgreSQL instance changes. With this patch, scrapeDSN won't
be run unless discoverDatabaseDSNs succeeded and thus the race condition is
eliminated.
Signed-off-by: Yann Soubeyrand <yann.soubeyrand@camptocamp.com>
Since we cannot know in advance the metrics which the exporter will generate,
the workaround is to run a Collect and return the metric descriptors. This is
problematic when the connection to the PostgreSQL instance cannot be
established straight from the start. This patch makes Describe return no
descriptors, effectively turning the collector in an unchecked one, which we're
in the typical use case here:
https://pkg.go.dev/github.com/prometheus/client_golang/prometheus?tab=doc#hdr-Custom_Collectors_and_constant_Metrics.
Signed-off-by: Yann Soubeyrand <yann.soubeyrand@camptocamp.com>
* Introduce histogram support
Prior to this change, the custom queries were restricted to counters and
gauges.
This change introduces a new ColumnUsage, namely HISTOGRAM, that expects
the column to contain an array of upper inclusive bounds for each
observation bucket in the emitted metric. It also expects three more
columns to be present with the suffixes:
- `_bucket`, containing an array of cumulative counters for the
observation buckets;
- `_sum`, the total sum of all observed values; and
- `_count`, the count of events that have been observed.
A flag has been added to the MetricMap struct to easily identify metrics
that should emit a histogram and the construction of a histogram metric
is aided by the pg.Array function and a new helper dbToUint64 function.
Finally, and example of usage is given in queries.yaml.
fixes#402
Signed-off-by: Corin Lawson <corin@responsight.com>
* Introduces tests for histogram support
Prior to this change, the histogram support was untested.
This change introduces a new integration test that reads a user query
containing a number of histogram metrics. Also, additional checks have
been added to TestBooleanConversionToValueAndString to test dbToUint64.
Signed-off-by: Corin Lawson <corin@responsight.com>
* do not panic when envs are set incorrectly
* do not panic when envs are set incorrectly - fix tests
Co-authored-by: Will Rouesnel <wrouesnel@wrouesnel.com>
The existing 'pg_stat_replication' data does not
include stats for inactive replication slots. This
commit adds a minimal amount of metrics from
'pg_replication_slots' to know if a slot is
active and its lag.
This is helpful to detect if an inactive slot
is causing the server to run out of storage due
to an inactive slot blocking WAL flushing.
Failures in parsing the user's queries are just being swallowed, which
makes troubleshooting YAML issues frustrating/impossible. I'm presuming
this was not intentional, since there is error handling code in the
function that calls this one, though it is unreachable as far as I can
tell without this change.
Co-authored-by: Will Rouesnel <wrouesnel@wrouesnel.com>
* Add a build info metric
Add a standard Prometheus build info metric to make monitoring rollouts
easier.
* Update prometheus vendoring.
* Fix build error
Fix missing bool in builtinMetricMaps.
Signed-off-by: Ben Kochie <superq@gmail.com>
During a refactor the pg_stat_bgwriter metrics were accidentally removed
and not re-added by 34fdb69ee2.
This commit restores these metrics. Resolves#336.
Closes#326 as is provides a viable solution to use a K8S init container
to fully contruct the PostgreSQL URI and 'hand it over' to the postgres_exporter
process.
In the user queries.yml file, the created namespaces can now be optionally
cached by setting cache_seconds, which will prevent the query being re-run
within that timeframe if previous results are available.
Supercedes #211, credit to @SamSaffron for the original PR.
* Fix problem: If autodiscovery is enable exporter make connection twice to database from connetion string (exclude current database from SQL-query);
* Fix problem: don't get default metrics and settings if autodiscovery is enabled. Now you can use --disable-default-metrics and --disable-settings-metrics with --auto-discover-databases and
* Use struct instead of interface{} when parsing query user
* Use MappingOptions
* Split function to be more testable
* Rename function to parseUserQueries
* Start to add test about query parsing
Some backstory
==============
I was attempting to use postgres_exporter with the official Docker
container (https://hub.docker.com/_/postgres) In a Kubernetes
StatefulSet, with a side-car configuration, but found that I wasn't able
to connect even with sharing the Postgres Unix listening socket, between
both containers. After copying the container over to an Alpine base I
quickly found out that the postgres_exporter was actually starting
before the main Postres container had dropped the unix socket onto the
file system, a quick work around is to write a bash for loop checking
for the existence of a unix socket, however this would require
maintaining a container, besides other users may find retries useful on
startup.
Implementation
==============
All changes are made to the getServer function and variables are
local, I was unsure if it was worth adding command line switches but
this would allow for a more sophisticated backOff loop in the future.
Hope this help, and let me know if you would like me to changes
anything.
If exporter is scraped by multiple Prometheuses (as we do) - Collect() could be called concurrently. In result in some cases one of Prometheuses could get pg_up = 0, because it was explicitly set to zero on first Collect call.
