Commit Graph

142745 Commits

Author SHA1 Message Date
Josh Salomon
6d27bbf77e osd: unit tests for read balance osd size aware score
Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>
2024-01-30 19:24:43 +00:00
Josh Salomon
c1cb08689f osd: Add score for read balance osd size aware policy
This score works for pools in which the read_ratio
value is set.
Current limitations:
- This mechanism ignores osd read affinty
  - There is a plan adding support for read affinity 0
    in the next version.
- This mechanism works only when all PGs are full
- If read_ration is not set - the existing mechanism (named
  fair score) is used.

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>
2024-01-30 19:24:43 +00:00
Josh Salomon
b13ffd2809 osd: Unit tests for read balancer on OSDs with different sizes
Test cases for the read balancer which takes osd sizes into account.
Some balancing code refactoring and reorg for code that is used in
multiple tests

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>
2024-01-30 19:24:43 +00:00
Josh Salomon
2953db8b58 osd: Read balancer for OSDs with different sizes
This commit adds calculation for desired primary distribution which
takes into account the osd size. This way smaller OSDs can take more
read operations (by adding more primaries) and the larger OSDs take less
primaries and the load of the cluater can increase. (This feature offset
a bit the weakest link in the chain effect under some conditions). In
order to calculate the loads correctly there is a need to know the
read/write ratio for the pool, and this commit assumes the read_ratio
parameter is available for the pool.

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>
2024-01-30 19:24:43 +00:00
Josh Salomon
8d7eb9f7e0 osd: Add 'read_ratio' pool parameterr
This parameter is used for better read balancing with non identical
devices.
- This parameter is controlled using the commands 'ceph osd pool set/get'
- This parameter is applicable only for replicated pools
- Valid values are integers in the range [0..100] and represent the
  percentage of read IOs out of all IOs in the pool
  - Value of 0 unsets this parameter and the value will be the default
    value (this is the generic behavior of the command 'ceph osd pool
    set'
  - default value can be set by config parameter
    `osd_pool_default_read_ratio`

Signed-off-by: Josh Salomon <41079547+JoshSalomon@users.noreply.github.com>
2024-01-30 19:24:43 +00:00
Laura Flores
faef8d54d8
Merge pull request #55364 from ljflores/wip-pr-triage
.github/workflows: disable sync-labels in pr-triage
2024-01-30 11:13:30 -06:00
Adam Kupczyk
1ef493c768
Merge pull request #53610 from Patty8122/documentation-update-fsck-flagdeep
doc/man/8: Add valid options in ceph-bluestore-tool.rst
2024-01-30 15:43:49 +01:00
Venky Shankar
8518c699e7 Merge PR #54515 into main
* refs/pull/54515/head:
	qa: get_global_id() returns a string (global_id)
	qa: correct usage of DEBUGFS_META_DIR in dedent

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2024-01-30 19:40:35 +05:30
Redouane Kachach
a44d457660
Merge pull request #55313 from rkachach/fix_issue_64183
mgr/rook: adding metrics monitoring e2e testing
2024-01-30 14:30:56 +01:00
Ivo Almeida
80d6ed3cb9
Merge pull request #55284 from rhcs-dashboard/cephfs-clone-create
mgr/dashboard: create cephfs snapshot clone
2024-01-30 08:30:51 -03:00
Rongqi Sun
774ce9f98b
Merge pull request #55351 from rosinL/wip-fix-64122
script/run-make: install lvm2 for make check cephadm test
2024-01-30 17:31:52 +08:00
Nizamudeen A
17b2100ffa
Merge pull request #55228 from rhcs-dashboard/fix-64081-main
mgr/dashboard: Export rgw multisite status via the API

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2024-01-30 14:39:02 +05:30
Venky Shankar
8f02b07290 Merge PR #52652 into main
* refs/pull/52652/head:
	PendingReleaseNotes: add note about new mdlog trimming configurations
	mds: drive mdlog trimming via a separate thread
	mds: allow runtime modification of mdlog trimming configuration
	mds: remove a bunch of heuristics from MDLog::trim()
	mds: add mdlog trimming threshold and decay counter

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
2024-01-30 13:10:19 +05:30
Venky Shankar
c9f71b12ed Merge PR #55184 into main
* refs/pull/55184/head:
	qa: remove redundant test test_volumes.TestRenameCmd.test_rename_when_fs_is_online

Reviewed-by: Christopher Hoffman <choffman@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2024-01-30 12:58:48 +05:30
Venky Shankar
0089c2a672 Merge PR #55346 into main
* refs/pull/55346/head:
	qa: remove error string checks and check w/ return value

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2024-01-30 12:26:32 +05:30
Nizamudeen A
a36bc781d3
Merge pull request #55349 from afreen23/fix-64112
mgr/dashboard: Create subvol of same name in different group

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2024-01-30 11:18:39 +05:30
Venky Shankar
e579ac55b5 PendingReleaseNotes: add note about new mdlog trimming configurations
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2024-01-30 10:58:48 +05:30
Venky Shankar
9c999db9c6 mds: drive mdlog trimming via a separate thread
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2024-01-30 10:58:48 +05:30
Venky Shankar
f9f3aef317 mds: allow runtime modification of mdlog trimming configuration
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2024-01-30 10:58:48 +05:30
Venky Shankar
3c3bc24ca3 mds: remove a bunch of heuristics from MDLog::trim()
These were probbaly introduced to workaround some sort of
resource overusage by the MDS during trimming, but now it
looks like they are not really neeeded, especially if we
introduce a dedicated thread for log trimming.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2024-01-30 10:58:48 +05:30
Venky Shankar
29610577ee mds: add mdlog trimming threshold and decay counter
Fixes: http://tracker.ceph.com/issues/61908
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2024-01-30 10:58:48 +05:30
Venky Shankar
7cd17931b1 qa: remove error string checks and check w/ return value
I ran into this failure once #54972 was merged. The test is validating
the error string returned due to the failed mount. There aren't any
return value checks - which is a _more_ important check. Generic error
string checks will fail once a (error) string is changed (typo, etc..).

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2024-01-30 10:24:59 +05:30
Venky Shankar
d45818bce9 Merge PR #54808 into main
* refs/pull/54808/head:
	client: fix copying bufferlist to iovec structures in Client::_read
	src/test: test sync call providing nullptr as ctx to async api

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Frank S. Filz <ffilzlnx@mindspring.com>
2024-01-30 10:03:57 +05:30
Venky Shankar
23443d3bc8 Merge PR #54031 into main
* refs/pull/54031/head:
	qa: add test to mangle lost+found directory object and ensure safety
	qa: run scrub before mounting client and validations

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
2024-01-30 09:59:57 +05:30
Nizamudeen A
e45f8eaf29
Merge pull request #55241 from rhcs-dashboard/remove-bucket-policy
mgr/dashboard: Remove bucket policy

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2024-01-30 09:42:19 +05:30
Aashish Sharma
b63e086fd5 mgr/dashboard: Export rgw multisite status via the API
Fixes: https://tracker.ceph.com/issues/64081

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
2024-01-30 09:00:49 +05:30
Laura Flores
6cca015822 .github/workflows: disable sync-labels in pr-triage
`sync-labels` is defined as "Whether or not to remove labels
when matching files are reverted or no longer changed by the PR".

We should disable this in case someone manually adds a label,
so the label doesn't get removed.

Signed-off-by: Laura Flores <lflores@ibm.com>
2024-01-29 17:47:22 -06:00
Laura Flores
7a04e9b9cf
Merge pull request #55326 from ceph/dependabot-github_actions-actions-labeler-5.0.0
.github: Bump actions/labeler from 4.0.2 to 5.0.0
2024-01-29 12:50:47 -06:00
Pedro Gonzalez Gomez
ae367352c7 mgr/dashboard: remove bucket policy
add a button on the bucket form to clear the bucket policy

Fixes: https://tracker.ceph.com/issues/64096
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
2024-01-29 19:06:24 +01:00
Pedro Gonzalez Gomez
83126556a8
Merge pull request #55062 from rhcs-dashboard/set-bucket-policy
mgr/dashboard: set bucket policies

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2024-01-29 18:39:45 +01:00
Nizamudeen A
e608006e7e
Merge pull request #54853 from ivoalmeida/snapshot-schedule-module
mgr/dashboard: snap schedule module detect/enable 

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2024-01-29 23:00:36 +05:30
Adam King
41652f8b41
Merge pull request #55255 from phlogistonjohn/jjm-mypy-updates
Various mypy related updates

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>
2024-01-29 11:27:34 -05:00
Casey Bodley
d7407c4ccb
Merge pull request #55329 from cbodley/wip-64189
rgw/rest: fix url decode of post params for iam/sts/sns

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
2024-01-29 16:11:52 +00:00
Pedro Gonzalez Gomez
2817d8e25d mgr/dashboard: set bucket policies
- Adds support to set bucket policies through the Dashboard.
- Rename rgw bucket policy from 'policy' to 'bucket policy' and tab 'Permissions' to 'Policies'
- Fix: hide Tags when none are present on bucket list details and sets bucket form dirty after deleting a tag
- Added service to manage the formatting of a textArea that works with json

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Fixes: https://tracker.ceph.com/issues/63942
2024-01-29 16:15:06 +01:00
Venky Shankar
f155b8fb61 Merge PR #54446 into main
* refs/pull/54446/head:
	qa: bump up scrub status command timeout

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2024-01-29 18:54:44 +05:30
Venky Shankar
527b465872 Merge PR #54683 into main
* refs/pull/54683/head:
	qa/tasks/cephfs/test_misc: switch duration to timeout
	qa/tasks/cephfs/test_misc: fix the incorrect comments

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2024-01-29 18:52:16 +05:30
Venky Shankar
6d61a73190 Merge PR #53578 into main
* refs/pull/53578/head:
	mds: adhere to mds_health_summarize_threshold config when reporting laggy clients

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2024-01-29 18:45:58 +05:30
Venky Shankar
320442d791 Merge PR #53177 into main
* refs/pull/53177/head:
	qa: test nfs export apply -i failure returns pseudo path in output JSON
	mgr/nfs: include pseudo in JSON output when nfs export apply -i fails

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
2024-01-29 18:44:28 +05:30
Venky Shankar
40cb741902 Merge PR #53734 into main
* refs/pull/53734/head:
	qa: refactor client upgrade yamls and other minor touchups
	qa/upgrade/nofs: upgrade pacific->reef
	qa/upgrade/upgraded_client: upgrade nautilus->pacific and pacific->reef

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2024-01-29 18:42:36 +05:30
Venky Shankar
52b20eb584 Merge PR #55309 into main
* refs/pull/55309/head:
	qa: `fs volume rename` requires `fs fail` and `refuse_client_session` set

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2024-01-29 18:41:23 +05:30
Redouane Kachach
21195114d4
mgr/rook: adding metrics monitoring e2e testing
Fixes: https://tracker.ceph.com/issues/64183

Signed-off-by: Redouane Kachach <rkachach@redhat.com>
2024-01-29 13:57:07 +01:00
Venky Shankar
1bf8291576 Merge PR #54451 into main
* refs/pull/54451/head:
	common/admin_socket: improvements to the RaiseHook

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2024-01-29 18:20:35 +05:30
luo rixin
d304cbb06e script/run-make: install lvm2 for make check cephadm test
The make check test `run-tox-cephdam` reports error:
```
        if errors:
>           raise Error('\nERROR: '.join(errors))
E           cephadmlib.exceptions.Error: lvcreate binary does not appear to be installed

cephadm.py:4434: Error
```
So let's installing lvm2 for make check cephadm test.

Fixes: https://tracker.ceph.com/issues/64122

Signed-off-by: luo rixin <luorixin@huawei.com>
2024-01-29 19:25:11 +08:00
Ivo Almeida
a57036a829 mgr/dashboard: snap schedule module detect/eneable
Fixes: https://tracker.ceph.com/issues/63768
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>
2024-01-29 10:59:42 +00:00
Ronen Friedman
6f6a7376f1
Merge pull request #55131 from ronen-fr/wip-rf-reserver
osd/scrub: partial implementation of scrub reserver
Reviewed-by: Samuel Just <sjust@redhat.com>-
2024-01-29 12:43:09 +02:00
Afreen
fb79df410d mgr/dashboard: Create subvol of same name in different group
Fixes https://tracker.ceph.com/issues/64112

Issue:
Currently, we are unable to create subvolume of same name in different
subvolume group

Fix:
We are validating only the filesystem name of subvolume
which is stopping the creation a subvolume of same name.
Added more granularity , by adding the subvolumegroup name.

Signed-off-by: Afreen <afreen23.git@gmail.com>
2024-01-29 15:42:10 +05:30
Matan Breizman
f63c485e60
Merge pull request #55296 from Matan-B/wip-matanb-crimson-pg-map-logs
crimson/osd/pg_map: Refactor

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2024-01-29 11:45:28 +02:00
Pedro Gonzalez Gomez
93c567e8ae
Merge pull request #55304 from afreen23/fix-64150
mgr/dashboard: Fix issue with Object's Overview page

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2024-01-29 08:56:55 +01:00
Nizamudeen A
c5367e9c4e
Merge pull request #54103 from ceph/dependabot-npm_and_yarn-src-pybind-mgr-dashboard-frontend-babel-traverse-7.23.2
mgr/dashboard: bump @babel/traverse from 7.23.0 to 7.23.2 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Nizamudeen A <nia@redhat.com>
2024-01-29 12:36:38 +05:30
Laura Flores
f45a39bdcd
Merge pull request #53988 from ljflores/wip-read-balancer-mgr-module 2024-01-28 23:40:01 -06:00