2021-08-20 16:19:26 +00:00
|
|
|
JAEGER- DISTRIBUTED TRACING
|
|
|
|
|
===========================
|
|
|
|
|
|
|
|
|
|
Jaeger provides ready to use tracing services for distributed
|
|
|
|
|
systems and is becoming the widely used standard because of their simplicity and
|
|
|
|
|
standardization.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
BASIC ARCHITECTURE AND TERMINOLOGY
|
|
|
|
|
----------------------------------
|
|
|
|
|
|
|
|
|
|
* TRACE: A trace shows the data/execution path through a system.
|
|
|
|
|
* SPAN: A single unit of a trace, it is a data structure that stores
|
|
|
|
|
information like operation name, timestamps, ordering in a trace.
|
|
|
|
|
* JAEGER CLIENT: language-specific implementations of the OpenTracing API.
|
|
|
|
|
* JAEGER AGENT: a daemon that listens for spans sent over User Datagram Protocol.
|
|
|
|
|
The agent is meant to be placed on the same host as the instrumented
|
|
|
|
|
application. (acts like a sidecar listener)
|
|
|
|
|
* JAEGER COLLECTOR: Jaeger agent sends the spans to this daemon which then
|
|
|
|
|
stitches the spans together to form a trace(if enabled, also persists a database
|
|
|
|
|
for these traces)
|
|
|
|
|
* JAEGER QUERY AND CONSOLE FRONTEND: UI based frontend to checkout the jaeger
|
|
|
|
|
traces, navigate to http://<jaeger frontend host>:16686
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
read more about jaeger tracing:.
|
|
|
|
|
|
2021-10-19 10:17:08 +00:00
|
|
|
https://www.jaegertracing.io/docs/
|
2021-08-20 16:19:26 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
JAEGER DEPLOYMENT
|
|
|
|
|
-----------------
|
|
|
|
|
|
|
|
|
|
there are couple of ways to deploy jaeger.
|
|
|
|
|
please refer to:
|
|
|
|
|
|
|
|
|
|
`jaeger deployment <https://www.jaegertracing.io/docs/1.25/deployment/>`_
|
|
|
|
|
|
|
|
|
|
`jaeger performance tuning <https://www.jaegertracing.io/docs/1.25/performance-tuning/>`_
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In addition, spans are being sent to local jaeger agent, so the jaeger agent must be running on each host (not in all-in-one mode).
|
|
|
|
|
otherwise, spans of hosts without active jaeger agent will be lost.
|
|
|
|
|
|
|
|
|
|
HOW TO ENABLE TRACING IN CEPH
|
|
|
|
|
-----------------------------
|
|
|
|
|
|
|
|
|
|
tracing in Ceph is disabled by default.
|
|
|
|
|
it could be enabled globally, or for each entity seperately (e.g. rgw).
|
|
|
|
|
|
|
|
|
|
Enable tracing globally::
|
|
|
|
|
|
|
|
|
|
$ ceph config set global jaeger_tracing_enable true
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Enable tracing for each entity::
|
|
|
|
|
|
|
|
|
|
$ ceph config set <entity> jaeger_tracing_enable true
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
TRACES IN RGW
|
|
|
|
|
-------------
|
|
|
|
|
|
|
|
|
|
traces of RGW can be found under Service `rgw` in Jaeger Frontend.
|
|
|
|
|
|
|
|
|
|
every user request is being traced. each trace contains tags for
|
|
|
|
|
`Operation name`, `User id`, `Object name` and `Bucket name`.
|
|
|
|
|
|
|
|
|
|
there is also `Upload id` tag for Multipart upload operations.
|
|
|
|
|
|
|
|
|
|
rgw service in Jaeger Frontend:
|
|
|
|
|
|
|
|
|
|
.. image:: ./rgw_jaeger.png
|
|
|
|
|
:width: 400
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
osd service in Jaeger Frontend:
|
|
|
|
|
|
|
|
|
|
.. image:: ./osd_jaeger.png
|
|
|
|
|
:width: 400
|
