Commit Graph

131138 Commits

Author SHA1 Message Date
Matan Breizman
9af99c3024 crimson/osd: Moving shard_services to PGBackend
ec and replicated backends are derived from PGBackend,
shard_services should be a member of the base class.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2022-05-10 14:16:05 +00:00
Casey Bodley
66331f4ba2
Merge pull request #46093 from pdvian/wip-admin-curl
examples: Add rgw script to access admin APIs using curl

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-05-10 08:24:41 -04:00
Ernesto Puerta
c959ae67ad
Merge pull request #46196 from rhcs-dashboard/smart-data-scsi
mgr/dashboard: smart data for devices with scsi protocol

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
2022-05-10 14:17:12 +02:00
Ernesto Puerta
7752d87712
Merge pull request #46201 from rhcs-dashboard/daemon_count_hosts
mgr/dashboard: show daemon count instead of services in host page 

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>
2022-05-10 13:52:18 +02:00
Ernesto Puerta
e24adc952b
Merge pull request #46174 from rhcs-dashboard/physical-disks-selection
mgr/dashboard: devices with same UID causes multiselection

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-05-10 13:51:27 +02:00
Ernesto Puerta
84f7da7ada
Merge pull request #46159 from rhcs-dashboard/disable-external-grafana-snapshots
mgr/cephadm: Disable external snapshots in grafana

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2022-05-10 13:48:19 +02:00
Soumya Koduri
450a6f0b41 rgw/dbstore: Support user creation via radosgw-admin
With the changes in https://github.com/ceph/ceph/pull/45987 ,
'radosgw-admin' command can be used to execute few admin operations on other stores.

This fix include changes to support  user creation/remove via `radosgw-admin`
command in dbstore.

Also fixed an issue with updating objv_tracker in op_state.user

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2022-05-10 15:36:08 +05:30
Venky Shankar
338e18b893
Merge pull request #46169 from rishabh-d-dave/vr-run-mnt-cmd
qa/vstart_runner: fix bug in LocalFuseMount._run_mount_cmd()

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-10 15:08:01 +05:30
Nizamudeen A
4fab663983 mgr/dashboard: smart data for devices with scsi protocol
In the dashboard, we've been showing smart data for hdd devices with ata
protocol only. Otherwise we show a No Smart Data found error which is
clearly misleading since Smart Data is returned even in the api call.

So this PR is trying to show the smart data for hdd devices
that uses scsi protocol too.

Fixes: https://tracker.ceph.com/issues/55574
Signed-off-by: Nizamudeen A <nia@redhat.com>
2022-05-10 12:55:18 +05:30
Nizamudeen A
ee78b47612 mgr/dashboard: show daemon count instead of services in host page
Fixes: https://tracker.ceph.com/issues/47218
Signed-off-by: Nizamudeen A <nia@redhat.com>
2022-05-10 12:14:33 +05:30
Rishabh Dave
c9807d35e0 qa/vstart_runner: fix args to os.path.join()
Result of os.path.join() before "./bin/ceph-mds" and after
"./bin/./ceph-mds".

Before -

	2022-05-05 19:36:11,100.100 DEBUG:__main__:> ./bin/./ceph-mds -i a

After -

	2022-05-05 19:38:48,179.179 DEBUG:__main__:> ./bin/ceph-mds -i a

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2022-05-10 11:03:24 +05:30
Rishabh Dave
3dd150959e qa/vstart_runner: minor bug fix and minor adjustments
The message regarding deletion of helper tools is printed for every
command. This message should be printed only when applicable.

Besides -

* Move XXX comments to _do_run() since it increases visibility of
  these messages.
* Move omission of arguments stuff to new method clear up the clutter.
* And remove shell as a parameter from _perform_checks_and_adjustments
  since it's redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2022-05-10 11:02:08 +05:30
Venky Shankar
f005520194
Merge pull request #37667 from rishabh-d-dave/vr-inherit-Remote
vstart_runner: let LocalRemote inherit Remote

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-10 10:30:22 +05:30
Samuel Just
ab6b68efe4
Merge pull request #46194 from athanatos/sjust/wip-55565
crimson: fix op tracking related bugs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-05-09 17:31:54 -07:00
Samuel Just
29a00b4482 crimson/common/operation: trim AggregateBlockingEvent::events entries
PglogBasedRecovery and BackfillRecovery reuse the same Operation
until their respective operations are complete.  Each recovery
operation adds an entry to AggregateBlockingEvent::events.  This
way, we only retain entries that are currently blocking.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-05-09 14:04:56 -07:00
Samuel Just
396ff7b702
Merge pull request #46127 from Matan-B/wip-matanb-cyan-clone
crimson/os: Add OP_CLONE to cyanstore

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-05-09 12:53:30 -07:00
Samuel Just
15b5a93a54
Merge pull request #46205 from rzarzynski/wip-crimson-cls_rgw-objapi
crimson/osd, rgw: implement the new Objclass API methods for cls_rgw

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-05-09 12:47:17 -07:00
Samuel Just
813ab9947d
Merge pull request #44418 from AmnonHanuhov/wip-fix_futurized_store
crimson/os: Don't limit the amount of returned keys per omap get call

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2022-05-09 12:46:25 -07:00
Radosław Zarzyński
86043d961d crimson/osd: drop "using std::string" from main.cc
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
2022-05-09 18:12:43 +02:00
Radosław Zarzyński
fc4e9ff852 crimson/osd: drop "using config_t = ..." from main.cc
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
2022-05-09 18:12:43 +02:00
Volker Theile
658486b566 mgr/dashboard: Creating and editing Prometheus AlertManager silences is buggy
When creating a new monitoring silence the form is pre-filled with the wrong alert data. It is always used the alert data from the very first object in the list of the API response but not the specified alert identified by the 'fingerprint' property.

The same problem applies to editing silences. The selected silence is not edited, it's always the first one in the list returned API response but not that with the specified 'id' property.

The main problem of the origin implementation is that the Prometheus Alertmanager API endpoints /api/v1/[alerts/silences] do not support querying. To fix that, filtering is done in the frontend.

Fixes: https://tracker.ceph.com/issues/55578

Signed-off-by: Volker Theile <vtheile@suse.com>
2022-05-09 17:25:34 +02:00
Radosław Zarzyński
14c04fb03d crimson/osd, rgw: implement the new Objclass API methods for cls_rgw
They were introduced by 3877c1e37f.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
2022-05-09 17:13:58 +02:00
Venky Shankar
d4f36d647c
Merge pull request #45732 from mchangir/mgr-snap_schedule-validate-fs-for-all-commands
mgr/snap_schedule: validate fs name argument

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-09 19:30:45 +05:30
dparmar18
c5f3c5122c cephfs-shell: make onecmd() print proper error msg
Rationale: Whenever a python exception occurred in cephfs-shell,
           it would often only be the exception message but doesn't
           say anything about the type of exception. For example if
           `ZeroDivisionError: division by zero` occurred, the onecmd()
           would print `division by zero` but will omit the type of
           exception. In this case it's easy to understand but let's
           say an `KeyError` exception occurred for a key `9999` which
           is not existent in the dictionary, onecmd() would print
           just `9999` in this scenario and it would be very difficult
           to interpret what type of error it is.

Fixes: https://tracker.ceph.com/issues/55536

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
2022-05-09 19:19:39 +05:30
Dhairya Parmar
53aa69e373
Merge branch 'master' into qa_testcephfs_shell_pep8_fixes 2022-05-09 15:46:04 +05:30
dparmar18
c02ab23ede qa/cephfs/test_cephfs_shell: Fixed pep8 formatting violations
NOTE: Although most of the issues are fixed but a few function
      and variable names are unchanged in order to prevent
      ambiguity and preserve their meaning.

      They are:
      - functions: setUp(), test_ls_H_prints_human_readable_file_size(),
      - variables: ls_H_output, ls_H_file_size

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
2022-05-09 15:40:08 +05:30
Nizamudeen A
f38e3e96b8 mgr/dashboard: fix smart data error
the error in the log was this

```
"/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 253, in _get_smart_data_by_device
May 06 07:38:39 occldlr750-1.occl208.lab conmon[2142938]:     svc_type, svc_id = daemon.split('.')
May 06 07:38:39 occldlr750-1.occl208.lab conmon[2142938]: ValueError: too many values to unpack (expected 2)
```

on the cluster, the output of `ceph device ls-by-host` looks like this

```
ceph: root@occldlr750-1 /]# ceph device ls-by-host occldlr750-1.occl208.lab
DEVICE                        DEV   DAEMONS                       EXPECTED FAILURE
DELLBOSS_VD_cbd004c975390010  sda   mon.occldlr750-1.occl208.lab
WDC_WUH721818AL5204_3FGZR3JT  sdda  osd.20
WDC_WUH721818AL5204_3FH4315T  sdbf  osd.94
WDC_WUH721818AL5204_3FHP58TT  sdec  osd.30
WDC_WUH721818AL5204_3FHSK8HT  sdu   osd.78
WDC_WUH721818AL5204_3FHVTS9T  sdfi  osd.47
WDC_WUH721818AL5204_3FHWJE8T  sdv   osd.23
WDC_WUH721818AL5204_3FHXHETT  sdcl  osd.11
WDC_WUH721818AL5204_3FHXKP1T  sdcj  osd.10
```

the first device is mon and its name is mon.occldlr750-1.occl208.lab.

In our dashboard code, when fetching the smart data we have a line like
this

`svc_type, svc_id = daemon.split('.')`

so for the mon the output of `daemon.split('.') will be ['mon', 'occldlr750-1', 'occl208', 'lab']. The svc_id gets split into three because of the split. I am changing that and giving the criteria as splitting only on the first occurence of the dot and the considering everything that comes after the dot as the svc_id of the device.

Fixes: https://tracker.ceph.com/issues/55571
Signed-off-by: Nizamudeen A <nia@redhat.com>
2022-05-09 11:14:49 +05:30
Liu-Chunmei
524ef52573
Merge pull request #46064 from liu-chunmei/crimson-pool-quota
Crimson pool quota

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-05-08 19:29:39 -07:00
Nizamudeen A
9fe18a6252 mgr/dashboard: devices with same UID causes multiselection
In the Physical Disks page, the uids for multiple devices are coming in
as same and that causes the selection to go berserk and select multiple
rows with same UID. The uid is generated in the frontend service call
itself. I just added some more parameters to it inorder to make it more
unique.

The second issue is the number of selected number getting multiplied
exponentially. Its because each time the table is updated or refreshed,
we push the row with the number of selected items we had before and that
causes the number of selection to multiply.

Fixes: https://tracker.ceph.com/issues/55523
Signed-off-by: Nizamudeen A <nia@redhat.com>
2022-05-08 20:20:35 +05:30
Matan Breizman
54b8d69232 crimson/os: Add OP_CLONE to cyanstore
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2022-05-08 08:27:25 +00:00
chunmei-liu
4b55a2b411 crimson/osd: check full
need check if pool exceeds its quota

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2022-05-07 16:47:33 -07:00
chunmei-liu
01e7122c97 crimson/errorator: change errorator template to int
need define EDQUOT which is not included in std::errc.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2022-05-07 16:36:21 -07:00
Samuel Just
89430b1c39 crimson: mark OSD::get_or_create_pg and wait_for_pg private
All users now use start_pg_operation.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-05-07 14:44:15 -07:00
Samuel Just
6a8bdb4bd1 crimson: refactor RepRequest to use start_pg_operation
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-05-07 14:44:15 -07:00
Samuel Just
04e2db38cf crimson: refactor RecoverySubRequest to use start_pg_operation
A side effect of this PR is that RecoverySubRequest will now use the
peering pipeline for ordering.  Not using before was, I think, actually
a bug which could result in pull operations ordering ahead of the
activation message when pulling objects from non-acting osds.

Signed-off-by: Samuel Just <sjust@redhat.com>
2022-05-07 14:44:11 -07:00
Samuel Just
d0708b5452 crimson/osd/pg_recovery: remove new_started, fix parallel_for_each call
Presumably typo from b7e2b11595.

Fixes: https://tracker.ceph.com/issues/55565
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-05-06 23:56:09 -07:00
Samuel Just
027d6c2940 crimson/common/operation: use std::list for AggregateBlockingEvent::events
std::vector's references aren't stable after emplace_back.

Fixes: https://tracker.ceph.com/issues/55565
Signed-off-by: Samuel Just <sjust@redhat.com>
2022-05-06 23:56:09 -07:00
Xuehan Xu
8c987827ad crimson/os/seastore/segment_cleaner: set gc size back to 1MB per cycle
This should reduce the conflicts between gc transactions and io transactions

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
09d971d25d crimson/os/seastore/btree: allow backref extents to be loaded from disk without initializing
For the purpose of higher space reclaim perf, we load backref extents by
parallelly invoking Cache::get_extent_by_type, which means loaded backref
extents won't be initialized

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
989138ef82 crimson/os/seastore/btree: fix log error
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
c64d6f5539 crimson/os/seastore: record replay_from info for dirty extents and alloc infos in segment header/tail
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
5059504fd1 crimson/os/seastore: better debug outputs
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
de4260d5ec crimson/os/seastore/transaction_manager: prepare extents for lba_manager/backref_manager's complete_transaction
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
ddcccb217f crimson/os/seastore/btree: clean up: add print_detail method for btree leaf nodes
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
17669171f6 crimson/os/seastore/segment_cleaner: trim journal and reclaim space with the help of backref
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
e79ad1a5aa crimson/os/seastore/transaction_manager: scan backref tree on mount
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
724a91c07f crimson/os/seastore/cache: record backref extents placement update in Cache::backref_extents
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
bf86258020 crimson/os/seastore: add depth parameter to FixedKVBtree::mapped_space_visitor_t
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
689c8ee2fd crimson/os/seastore/cache: replay alloc info deltas
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00
Xuehan Xu
ed9c7734d6 crimson/os/seastore/segment_cleaener: flush backrefs when trimming journal
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-07 13:13:38 +08:00