4.9 KiB
Secure Alertmanager cluster traffic
Type: Design document
Date: 2019-02-21
Author: Max Inden IndenML@gmail.com
Status Quo
Alertmanager supports high availability by interconnecting multiple Alertmanager instances building an Alertmanager cluster. Instances of a cluster communicate on top of a gossip protocol managed via Hashicorps Memberlist library. Memberlist uses two channels to communicate: TCP for reliable and UDP for best-effort communication.
Alertmanager instances use the gossip layer to:
- Keep track of membership
- Replicate silence creation, update and deletion
- Replicate notification log
As of today the communication between Alertmanager instances in a cluster is sent in clear-text.
Goal
Instances in a cluster should communicate among each other in a secure fashion. Alertmanager should guarantee confidentiality, integrity and client authenticity for each message touching the wire. While this would improve the security of single datacenter deployments, one could see this as a necessity for wide-area-network deployments.
Non-Goal
Even though solutions might also be applicable to the API endpoints exposed by Alertmanager, it is not the goal of this design document to secure the API endpoints.
Proposed Solution - TLS Memberlist
Memberlist enables users to implement their own transport layer without the need of forking the library itself. That transport layer needs to support reliable as well as best-effort communication. Instead of using TCP and UDP like the default transport layer of Memberlist, the suggestion is to only use TCP for both reliable as well as best-effort communication. On top of that TCP layer, one can use mutual TLS to secure all communication. A proof-of-concept implementation can be found here: https://github.com/mxinden/memberlist-tls-transport.
The data gossiped between instances does not have a low-latency requirement that TCP could not fulfill, same would apply for the relatively low data throughput requirements of Alertmanager.
TCP connections could be kept alive beyond a single message to reduce latency as well as handshake overhead costs. While this is feasible in a 3-instance Alertmanager cluster, the discussed custom implementation would need to limit the amount of open connections for clusters with many instances (#connections = n*(n-1)/2).
As of today, Alertmanager already forces Memberlist to use the reliable TCP instead of the best-effort UDP connection to gossip large notification logs and silences between instances. The reason is, that those packets would otherwise exceed the MTU of most UDP setups. Splitting packets is not supported by Memberlist and was not considered worth the effort to be implemented in Alertmanager either. For more info see this Github issue.
With the last Prometheus developer summit in mind, the Prometheus projects preferred security mechanism seems to be mutual TLS. Having Alertmanager use the same mechanism would ease deployment with the rest of the Prometheus stack.
As a side effect (benefit) Alertmanager would only need a single open port (TCP traffic) instead of two open ports (TCP and UDP traffic) for cluster communication. This does not affect the API endpoint which remains a separate TCP port.
Alternative Solutions
Symmetric Memberlist
Memberlist supports symmetric key encryption via AES-128, AES-192 or AES-256 ciphers. One can specify multiple keys for rolling updates. Securing the cluster traffic via symmetric encryption would just involve small configuration changes in the Alertmanager code base.
Replace Memberlist
Coordinating membership might not be required by the Alertmanager cluster component. Instead this could be bound to static configuration or e.g. DNS service discovery. On the other hand, gossiping silences and notifications is ideally done in an eventual consistent gossip fashion, given that Alertmanager is supposed to scale beyond a 3-instance cluster and beyond local-area-network deployments. With these requirements in mind, replacing Memberlist with an entirely self-built communication layer is a great undertaking.
TLS Memberlist with DTLS
Instead of redirecting all best-effort traffic via the reliable channel as proposed above, one could also secure the best-effort channel itself using UDP and DTLS in addition to securing the reliable traffic via TCP and TLS. DTLS is not supported by the Golang standard library.