Commit Graph

129108 Commits

Author SHA1 Message Date
Casey Bodley
fb90d0ebdf build: revert arrow package dependency
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2022-01-14 14:54:09 -05:00
Casey Bodley
077e234b74 cmake: disable parquet by default
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2022-01-14 14:50:47 -05:00
Ernesto Puerta
581e10f9e8
Merge pull request #44523 from ljflores/wip-telemetry-dashboard
mgr/dashboard/telemetry: reduce telemetry dashboard preview size

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: neha-ojha <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
2022-01-14 20:11:15 +01:00
Yuri Weinstein
8b1568832d
Merge pull request #44550 from jdurgin/wip-pool-get-quota
mon/OSDMonitor: avoid null dereference if stats are not available

Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-01-14 10:46:49 -08:00
Yuri Weinstein
e419a29be5
Merge pull request #42735 from amathuria/wip-amathuria-scrub-stats
osd/scrub: Add stats to PG dump for number of objects scrubbed

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2022-01-14 10:46:28 -08:00
Neha Ojha
82ff85871e
Merge pull request #43667 from ifed01/wip-ifed-fix-ram-gridy-fsck
os/bluestore: make shared blob fsck much less RAM-greedy.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
2022-01-14 10:27:31 -08:00
Soumya Koduri
0bd3c05c95
Merge pull request #44440 from soumyakoduri/wip-skoduri-dbstore-fixes
rgw/dbstore: Misc fixes
2022-01-14 23:38:22 +05:30
Neha Ojha
ed7fe35439
Merge pull request #44552 from jdurgin/wip-releases-doc
doc/releases: remove outdated info and versions; mark nautilus eol

Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-01-14 09:42:08 -08:00
Yuri Weinstein
755dc55b8e
Merge pull request #44370 from benhanokh/NCB_expand_device_fix
NCB code doesn't update allocation file when we expand-device

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
2022-01-14 09:06:41 -08:00
Yuri Weinstein
f5b4f3f4d9
Merge pull request #44251 from yaarith/telemetry-opt-in
mgr/telemetry: introduce new design for varying report data

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2022-01-14 09:06:11 -08:00
Yuri Weinstein
31ff4958bf
Merge pull request #43849 from rzarzynski/wip-bs-lucky-buffers
blk, os/bluestore: introduce huge page-based read buffers

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2022-01-14 08:44:06 -08:00
Radoslaw Zarzynski
fba106f172
Merge pull request #42576 from AmnonHanuhov/wip-port_rgw_classes
crimson/osd: Port rgw object classes to run in crimson

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-01-14 16:48:42 +01:00
Yuri Weinstein
c8ea20345d
Merge pull request #44518 from gregsfortytwo/wip-fix-53824
osd: PeeringState: fix selection order in calc_replicated_acting_stretch

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2022-01-14 07:47:00 -08:00
Laura Flores
68df650818 mgr/dashboard/telemetry: add test for formatReport()
Tests a scenario where all keys are removed, and one
where a key is ignored.

Signed-off-by: Laura Flores <lflores@redhat.com>
2022-01-14 14:37:10 +00:00
Aishwarya Mathuria
0484c76b73 doc: document new OBJECTS_SCRUBBED column in pg dump
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
2022-01-14 19:40:33 +05:30
Amnon Hanuhov
b51f2e04e7 crimson/osd: Implement missing objclass functions used by cls_rgw
Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>
2022-01-14 14:13:14 +02:00
Amnon Hanuhov
e816237374 crimson/osd: Add support for CEPH_OSD_OP_OMAPRMKEYS
Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>
2022-01-14 14:13:14 +02:00
Amnon Hanuhov
d04d2ef025 crimson/osd: Add a getter for last_user_version
last_user_version is the last user object version applied to store

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>
2022-01-14 14:13:14 +02:00
Amnon Hanuhov
217d0e19dc crimson/osd: drop PGBackend& from OpsExecuter ctor
OpsExecuter holds a Ref<PG> so the PGBackend can be extracted from it
using get_backend()

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>
2022-01-14 14:13:13 +02:00
Amnon Hanuhov
59d6f95c70 crimson/osd: drop pg_pool_t from OpsExecuter ctor
OpsExecuter now holds a Ref<PG> so the pool info can be extracted from it
using get_pool().info

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>
2022-01-14 14:13:13 +02:00
Amnon Hanuhov
56bf9df993 crimson/osd: Store a reference to PG inside OpsExecuter
This is needed as some ObjClass methods make use of pg information related to the given cls_method_context_t

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>
2022-01-14 14:13:13 +02:00
Ernesto Puerta
4def95b0d4
Merge pull request #44507 from votdev/issue_53813_nfs_page_not_found
mgr/dashboard: NFS pages shows 'Page not found'

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2022-01-14 12:56:55 +01:00
Ernesto Puerta
f7cd4b2873
Merge pull request #43685 from p-se/fix-grafana-graphs-ceph_daemon
mgr/dashboard: fix Grafana OSD/host panels

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-01-14 12:50:13 +01:00
Ernesto Puerta
c208dbeb13
Merge pull request #44573 from rhcs-dashboard/53858-fix-smart-data-single-daemon
mgr/dashboard: fix: get SMART data from single-daemon device

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-01-14 12:48:52 +01:00
Ilya Dryomov
d665f88ad0
Merge pull request #44559 from ideepika/wip-iscsi-53830
test/rbd/iscsi: correct the hostname in gwcli_create.t to match hostname -f

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2022-01-14 10:30:27 +01:00
Ilya Dryomov
3c2b05a252
Merge pull request #44571 from idryomov/wip-xfstests-qemu-cert
qa/run_xfstests_qemu.sh: stop reporting success without actually running any tests

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2022-01-14 10:28:06 +01:00
Venky Shankar
e65d88ca58
Merge pull request #44570 from vshankar/wip-53857
qa: adjust for MDSs to get deployed before verifying their availability

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-01-14 08:42:20 +05:30
Samuel Just
2389ebd7d4
Merge pull request #44555 from cyx1231st/wip-fix-seastore-jounral-fast-submit
crimson/os/seastore/journal: fast submit if RecordSubmitter is IDLE and no pending

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2022-01-13 17:23:37 -08:00
Josh Durgin
5ecc3d9776 doc/releases: remove dev and pre-nautilus releases from timeline
Improve readability of the table - all this information is still
preserved in older branches.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2022-01-13 19:52:20 -05:00
Adam King
07ff5bcc55
Merge pull request #44583 from mgfritch/fixup-44306-docker-count
cephadm: increase number of docker.io occurances

Reviewed-by: Adam King <adking@redhat.com>
2022-01-13 19:18:35 -05:00
Michael Fritch
b0b5214b8f
cephadm: increase number of docker.io occurances
fixup for 0fe2e54db774271e4fc18b45aba36b66cbc71779

Signed-off-by: Michael Fritch <mfritch@suse.com>
2022-01-13 15:52:09 -07:00
Yaarit Hatuka
2197f7ee01 mgr/telemetry: revise format_perf_histogram
osd_perf_histograms now include only separated stats; remove the
aggregated formatting; we can revert this in case we ever add aggregated
histograms.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
30c4944938 PendingReleaseNotes: add a note about telemetry
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
2d1550cf05 mgr/telemetry: add enable / disable channel all
Enable or disable all telemetry channels at once with:
    ceph telemetry enable channel all
    ceph telemetry disable channel all

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
71b5aa7d3b mgr/telemetry: do not restore channels default when opting-out
Other modules do not reset their configuration; keep telemetry module
consistent with this behavior.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
813ab8a7c6 mgr/telemetry: verify there are new collections when nagging due to a major
upgrade

When adding a new collection we define whether to nag the user about it.
We may add many collections and nag about none of them. However, in case
of a major upgrade, we wish to notify the user about these new
collections. This commit verifies there are indeed new collections when
nagging due to a major upgrade.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
77a032526d mgr/telemetry: improve output of ceph telemetry collection ls
STATUS column now indicates whether a collection is being reported, and
the reasons why it's not (either the user is not opted-in to this
collection, or its channel is off).

Also, removed the ENROLLED and DEFAULT columns due to potential
confusion they may cause.

In case a user is not opted-in to certain collections, a message will
appear above the table with the missing collections:

    New collections are available:
    ['basic_base', 'basic_mds_metadata', 'crash_base', 'device_base',
    'ident_base', 'perf_perf']
    Run `ceph telemetry on` to opt-in to these collections.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
b651619aff mgr/telemetry: use dict lookup when traversing MODULE_COLLECTION
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
3bd87ea644 mgr/telemetry: add test coverage for telemetry upgrade
Test the behavior of the module after an upgrade, as we shift from our
revision design to Collections.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:54:07 +00:00
Yaarit Hatuka
4c110ed2a5 doc/mgr/telemetry: document new commands
New commands:

  ceph telemetry enable channel <channel_name>
  ceph telemetry disable channel <channel_name>
  ceph telemetry channel ls
  ceph telemetry collection ls
  ceph telemetry collection diff
  ceph telemetry preview
  ceph telemetry preview-device
  ceph telemetry preview-all

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:53:47 +00:00
Yaarit Hatuka
7241fe96eb mgr/telemetry: add command to list all collections
List all collections, their current enrollment state, status, default,
and description, with:

$ ceph telemetry collection ls

NAME                  ENROLLED    STATUS    DEFAULT    DESC
basic_base            TRUE        ON        ON         Basic information about the cluster (capacity, number and type of daemons, version, etc.)
basic_mds_metadata    TRUE        ON        ON         MDS metadata
crash_base            TRUE        ON        ON         Information about daemon crashes (daemon type and version, backtrace, etc.)
device_base           TRUE        ON        ON         Information about device health metrics
ident_base            TRUE        OFF       OFF        User-provided identifying information about the cluster
perf_perf             TRUE        OFF       OFF        Information about performance counters of the cluster

Please note:

NAME:
=====
Collection name; prefix indicates the channel the collection belongs to.

ENROLLED:
=========
Signifies the collections that were available in the module when the
user last opted-in to telemetry. Please note: Even if a collection is
'enrolled', its metrics will be reported only if its channel is enabled.

STATUS:
=======
Indicates whether the collection metrics are reported; this is
determined by the status (enabled / disabled) of the channel the
collection belongs to, along with the enrollment status of the
collection.

DEFAULT:
========
The default status (enabled / disabled) of the channel the collection
belongs to.

DESC:
=====
Collection description.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:53:47 +00:00
Yaarit Hatuka
7862046c42 mgr/telemetry: fix missing type annotations
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:53:47 +00:00
Yaarit Hatuka
dbe8500a78 mgr/telemetry: add preview-device and preview-all commands
`ceph telemetry show` will show a sample cluster report if the user is
opted-in to telemetry. The report will be compiled of the collections
the user is opted-in to. To preview a report compiled of the most recent
collection available, use `ceph telemetry preview`.

The device channel is not included in the cluster report, since it's
being sent to a different endpoint, thus we use
`ceph telemetry show-device` in case the user is opted-in to telemetry
and the device channel is enabled. If not, it can also be previewed with
`ceph telemetry preview-device`.

If telemetry is on, and device channel is enabled, both reports can be
reviewed with `ceph telemetry show-all`, otherwise use
`ceph telemetry preview-all`.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:53:47 +00:00
Yaarit Hatuka
1c1fcad510 mgr/telemetry: add command to list all channels
List all channels, their current state, default, and description, with:

$ ceph telemetry channel ls

NAME      ENABLED    DEFAULT    DESC
basic     ON         ON         Share basic cluster information (size, version)
ident     OFF        OFF        Share a user-provided description and/or contact email for the cluster
crash     ON         ON         Share metadata about Ceph daemon crashes (version, stack straces, etc)
device    ON         ON         Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)
perf      ON         OFF        Share perf counter metrics summed across the whole cluster

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:53:47 +00:00
Yaarit Hatuka
012bde7b26 mgr/telemetry: add commands to enable/disable channels
Currently we enable/disable a telemetry channel via CLI with:
  `ceph config set mgr mgr/telemetry/channel_basic true`
  `ceph config set mgr mgr/telemetry/channel_crash false`

We can now do this with:
  `ceph telemetry enable channel basic`
  `ceph telemetry disable channel crash`

We allow enabling / disabling lists of channels:
  `ceph telemetry enable channel basic device crash perf`
  `ceph telemetry disable channel basic device crash perf`

Please note, telemetry should be on for these commands to take effect.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:53:47 +00:00
Yaarit Hatuka
0e1909417c mgr/telemetry: introduce new design for adding new data
The current design requires increasing the telemetry revision each time
we add new data to the report. As a result, users need to re-opt-in to
telemetry. This new design allows for adding new data to the report,
while allowing users to keep sending only what they already opted-in to,
hence no re-opt-in is required. In case users wish to report the new
data as well, they need to re-opt-in and enable any new channels.

Also, move formatting perf histograms to a function, so we can use it
both in `show` and `preview` commands.

Fix get_report call in dashboard to use get_report_locked.

Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2022-01-13 21:51:57 +00:00
Josh Durgin
bd05cede10
Merge pull request #44554 from jdurgin/wip-rbd-qos-docs
doc/rbd: clarify and add more detail to librbd QoS docs

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2022-01-13 12:02:03 -08:00
Casey Bodley
f65b59be85
Merge pull request #40802 from galsalomon66/wip-s3select-parquet-object-processing-2
RGW/s3select : parquet implementation:

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-01-13 12:53:33 -05:00
Deepika Upadhyay
8416173a7b test/rbd/iscsi: correct the HOST name provided.
hostname -f and hostname generated from gwcli_create being different
gave rise to error:

The first gateway defined must be the local machine

Fixes: https://tracker.ceph.com/issues/53830

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2022-01-13 22:49:28 +05:30
Kefu Chai
40717bf5a9
Merge pull request #44577 from clementperon/master
cmake: Fix Finddpdk cmake module

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-01-14 01:15:07 +08:00