This adds metrics that look like this:
```
alertmanager_alerts{state="active"} 6
alertmanager_alerts{state="suppressed"} 0
alertmanager_silences{state="active"} 1
alertmanager_silences{state="expired"} 1
alertmanager_silences{state="pending"} 0
```
This can be used to monitor alertmanager's usage and validate that
alertmanagers in a mesh have a similar number of silences and alerts.
* silences: avoid deadlock
Calling gossip.GossipBroadcast() will cause a deadlock if
there is a currently executing OnBroadcast* function.
See #982
* silence_test: better unit test to detect deadlocks
This fixes#559 by removing concurrent map writes to the matcher cache.
The cache was guarded by the Silence's main lock, which only used a
read-lock on queries.
The cache's get methods lazily loads data into the cache and thus
causing concurrent writes.
We just change the main lock to always write-lock, as we don't expect
high lock contention at this point and would have it in a dedicated
cache lock anyway.
This commit adds an implementation of a silence storage that can
share store and modify silences, share state via a mesh network,
write and load snapshots, and be dynamically queried.
All data formats are based on protocol buffers.
