Commit Graph

129608 Commits

Author SHA1 Message Date
Ali Maredia
8de2884e06
Merge pull request #44856 from cbodley/wip-54103
qa/rgw: install s3cmd with extra_system_packages

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2022-02-07 16:02:40 -05:00
Ilya Dryomov
56e348e4ad
Merge pull request #44886 from holzman/bugfix-54128
krbd: return error when no initial monitor address found

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2022-02-07 21:41:06 +01:00
Burt Holzman
0076ffc86e krbd: return error when no initial monitor address found
Since we filter monitor addresses based on ms_mode, check that at
least one address was found.

Otherwise, we mismatch arguments when calling sysfs/add_single_major
which emits a misleading error message to dmesg:

  libceph: resolve 'name=user1' (ret=-3): failed
  libceph: parse_ips bad ip 'name=user1,key=client.user1'

Fixes: https://tracker.ceph.com/issues/54128

Signed-off-by: Burt Holzman <burt@fnal.gov>
2022-02-07 10:47:31 -06:00
Daniel Gryniewicz
b82e3e68a9 RGW - Move early return
When transitioning an object to cloud, there was an early return,
skipping the removal of the cloud target.  Fix this to be in the right
place.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2022-02-07 10:08:02 -05:00
Ernesto Puerta
80ae7c307e
Merge pull request #44825 from rhcs-dashboard/rgw-dashboard-test-hanging
mgr/dashboard: fix for cephadm e2e failing because of rgw commands getting stuck

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-02-07 14:07:43 +01:00
Ernesto Puerta
792a03d7b1
Merge pull request #44920 from rhcs-dashboard/cephadm-bootstrap-fix
cephadm: change shared_folder directory for prometheus and grafana

Reviewed-by: MrFreezeex <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-02-07 14:02:20 +01:00
Yingxin Cheng
614088b684 crimson/os/seastore/lba_manager: convert to seastore logging
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-02-07 20:09:05 +08:00
Nizamudeen A
27592b7561 cephadm: change shared_folder directory for prometheus and grafana
After https://github.com/ceph/ceph/pull/44059 the monitoring/prometheus
and monitoring/grafana/dashboards directories are changed to
monitoring/ceph-mixins. That broke the shared_folders in the cephadm
bootstrap script.

Changed all the instances of monitoring/prometheus and
monitoring/grafana/dashboards to monitoring/ceph-mixins

Also, renaming all the instances of prometheus_alerts.yaml to
prometheus_alerts.yml.

Fixes: https://tracker.ceph.com/issues/54176
Signed-off-by: Nizamudeen A <nia@redhat.com>
2022-02-07 16:34:37 +05:30
Yingxin Cheng
c277fdecc5 crimson/os/seastore/lba_manager: move lba_tree_inner_stats into LBAManager
Cleanup, change stats from LBATree static member to BtreeLBAManager
class member.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-02-07 18:58:35 +08:00
yuliyang_yewu
3990337fad rgw: fix dbstore test init env wrong
fix https://tracker.ceph.com/issues/54174

Signed-off-by: yuliyang_yewu <yuliyang_yewu@cmss.chinamobile.com>
2022-02-07 14:39:38 +08:00
Venky Shankar
32faa6b2f5
Merge pull request #44503 from lxbsz/typo1
mds: fix a typo of inode version related comment

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-02-07 10:03:31 +05:30
Soumya Koduri
a5627d6039 rgw/dbstore: Create GC Thread to delete stale objects
Create a gc thread to cleanup the stale tail objects data

XXX: handle read + delete usecase, simple approach could be
to use locks or sqlite transactions in GC

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2022-02-07 07:39:55 +05:30
Soumya Koduri
38052496c8 rgw/dbstore: Use Object ID to handle racing writes
Create unique ID for each object upload which will be
atomically updated in the head object at the end. This will
prevent data corruption during concurrent writes.
Incase of Multipart Uploads, upload_id is used as ObjectID.

XXX: The stale or obsolete tail data needs to be deleted

Also addressed invalid usage of CephContext in dbstore tests.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2022-02-07 07:39:55 +05:30
Vicente Cheng
62f0140d80 test/msgr: add unittest to simulate network block temporarily
Add new test case to verify the network block temporarily,
   that case would make outgoing_bl overflow so add the assert
   checking mechanism to claim_append

   Just use 2 connections because that we could not generate the
   large data set to verify it

   Simulate the EAGAIN situation looks like by skip calling
   cs.send() because EAGAIN would return size 0 and keep the
   outgoing_bl

Signed-off-by: Vicente Cheng <vicente_cheng@bigtera.com>
2022-02-07 02:09:38 +00:00
Ilya Dryomov
bad21fa497
Merge pull request #44842 from idryomov/wip-krbd-rxbounce-option
rbd: recognize rxbounce map option

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2022-02-06 20:37:31 +01:00
Ilya Dryomov
512f8da720
Merge pull request #43910 from sunnyku/wip-rbd-snap-sched
mgr/rbd_support: fix schedule remove

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2022-02-06 20:32:09 +01:00
Matan Breizman
6455ee18a4 test/crimson: inject Socket layer delays/failures msgr utest
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2022-02-06 08:45:49 +00:00
Casey Bodley
1d25e87ed3
Merge pull request #44893 from cfsnyder/wip-54130-oplog-segfault
rgw: fix segfault in OpsLogRados::log when realm is reloaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-02-05 08:54:57 -05:00
Venky Shankar
0e37e6a7be
Merge pull request #44851 from batrick/i54081
mds: add inline flag to MDS bootstrap compat in MDSMap

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-02-05 18:04:29 +05:30
Samuel Just
83200ce754
Merge pull request #44887 from josephsawaya/vstart-dev
src/vstart.sh: Add --no-restart to vstart & add mkfs logs to osd.stdout

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-02-04 21:26:11 -08:00
wanghao72
9d8093311a rgw: CopyObject works with x-amz-copy-source-if-* headers
CopyObject api support condition headers, eg x-amz-copy-source-if-match, while radosgw miss out the 'source' keyword
Fixes: https://tracker.ceph.com/issues/53945

Signed-off-by: Wang Hao <wanghao72@baidu.com>
2022-02-05 12:07:45 +08:00
Joseph Sawaya
fc3b6273f3 src/vstart.sh: Add --no-restart to vstart & add mkfs logs to osd.stdout
The output of mkfs wasn't also being included in the OSD's log before this which
can make it more difficult to debug issues with mkfs.

ceph-run restarting every 5 seconds can make it difficult to read the osd's stdout.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
2022-02-04 16:07:35 -05:00
Neha Ojha
8a8945e640
Merge pull request #44868 from neha-ojha/wip-move-to-stream
qa/distros: remove centos8

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2022-02-04 11:56:08 -08:00
Casey Bodley
f692731e2a
Merge pull request #44866 from yuvalif/wip-yuval-fix-datalog-list
rgw/admin: fix radosgw-admin datalog list max-entries issue

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2022-02-04 14:55:07 -05:00
Casey Bodley
ca8e0a64b1
Merge pull request #44386 from ivancich/wip-reshard-tentative-goal
rgw: in bucket reshard list, clarify new num shards is tentative

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
2022-02-04 14:54:36 -05:00
Casey Bodley
235b51b786
Merge pull request #42330 from SWJTU-ZhangLei/wip-rgw-postobj
rgw: RGWPostObj::execute() may lost data.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-02-04 14:50:45 -05:00
Casey Bodley
425100334b
Merge pull request #44743 from linuxbox2/wip-rgwlc-lcgetv
rgwlc: fix compat-decoding of cls_rgw_lc_get_entry_ret

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-02-04 14:50:27 -05:00
Casey Bodley
70630b9644
Merge pull request #44137 from ivancich/wip-fix-reshard-cancel
rgw: "reshard cancel" errors with "invalid argument"

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-02-04 14:50:10 -05:00
Casey Bodley
1762305bb4
Merge pull request #44892 from trociny/wip-54119
rgw: check bucket shard init status in RGWRadosBILogTrimCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-02-04 14:49:47 -05:00
Ilya Dryomov
fbf8c1d68b qa/suites/krbd: add legacy+rxbounce and crc+rxbounce coverage
For basic, rbd and rbd-nomount subsuites, replace legacy and crc
facets with "legacy or legacy+rxbounce" and "crc or crc+rxbounce"
facets (chosen at random).

For fsx, singleton and thrash subsuites, add legacy+rxbounce and
crc+rxbounce facets and drop prefer-crc facet.  The expected behaviour
of the latter depends on cluster configuration and should be tested
separately.

The total number of jobs remains the same.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-02-04 19:04:38 +01:00
Ilya Dryomov
95d30b534e qa: krbd rxbounce test
Lives in its own directory since ms_mode doesn't need to be permuted
here.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-02-04 19:04:37 +01:00
Ilya Dryomov
8d2a456d70 rbd: recognize rxbounce map option
Fixes: https://tracker.ceph.com/issues/54063
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-02-04 19:04:37 +01:00
Neha Ojha
e84992348e
Merge pull request #44563 from benhanokh/NCB_new_alloc_map
BlueStore:NCB:Bug-Fix for recovery code with shared blobs

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
2022-02-04 09:31:16 -08:00
Ernesto Puerta
6a4b1e148d
Merge pull request #44796 from pcuzner/remove-old-mib
monitoring: remove old MIB

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-02-04 17:42:08 +01:00
Ernesto Puerta
63f5fa45fe
Merge pull request #44857 from rhcs-dashboard/test-doc-dash-cephadm
doc: update kcli test env documentation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-02-04 17:35:54 +01:00
Ernesto Puerta
84287bdbb6
Merge pull request #44059 from MrFreezeex/mixins
mgr/dashboard: monitoring: refactor into ceph-mixin

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2022-02-04 17:28:44 +01:00
Daniel Gryniewicz
a7e8d68d5e
Merge pull request #44862 from andriytk/fix-put-memleak2
rgw_sal_motr: fix possible memleak on put
2022-02-04 11:24:58 -05:00
Sunny Kumar
a9312d4777 qa/workunits/rbd: improve schedule add/remove cli test
This patch adds few tests to cover schedule add/remove with invalid
inputs.

Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
2022-02-04 16:16:03 +00:00
Sunny Kumar
1b62447071 mgr/rbd_support: fix schedule remove
Issue:

If we provide a random string in the schedule remove
command the entire schedule at specified level gets
removed.

Fixes: https://tracker.ceph.com/issues/53250
Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
2022-02-04 16:15:30 +00:00
Casey Bodley
a2d2ed2d8e
Merge pull request #44348 from soumyakoduri/wip-skoduri-lc-teuthology
rgw/qa: Add test suite for lifecycle cases

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-02-04 11:04:13 -05:00
Nizamudeen A
347fb2e8fe mgr/dashboard: fix for cephadm e2e failing because of rgw commands getting stuck
Delaying the rgw service creation in the tests until the cluster is
healthy

also changing the node_ip_offset to 110 because in the jenkins I saw

Fixes: https://tracker.ceph.com/issues/54030
Signed-off-by: Nizamudeen A <nia@redhat.com>
2022-02-04 14:31:22 +05:30
Gabriel BenHanokh
8868894491 os/BlueStore: NCB fixes recovery code with shared blobs
Replaces the BitmapAllocator used by NCB Recovery code with a dedicated SimpleBitmap.
The SimpleBitmap allows for bits to be set multiple times without any adverse effect.
This is needed beacuse shared-blobs will report the same allocation multiple times.

Fixes: https://tracker.ceph.com/issues/53678

Signed-off-by: Gabriel Benhanokh <gbenhano@redhat.com>
2022-02-04 07:35:31 +02:00
Neha Ojha
c2a3676b29
Merge pull request #44003 from zdover23/wip-doc-2021-11-17-start-documenting-ceph-python-pip-update
doc/start: update python-pip to python3-pip

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-02-03 15:36:22 -08:00
Neha Ojha
630b43d851
Merge pull request #44560 from jdurgin/wip-edit-on-github
doc/conf.py: use links to the 'edit' page for edit on github

Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-02-03 15:28:16 -08:00
Samuel Just
2b955b9e3b
Merge pull request #44888 from josephsawaya/zns-dev
crimson/os/seastore: change segment_off_t to seastore_off_t in ZNSSegmentManager

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-02-03 15:00:47 -08:00
Patrick Donnelly
4fbd42ab5b
Merge PR #44691 into master
* refs/pull/44691/head:
	doc: document PI .qa symlinks
	qa: fix or add missing .qa links
	qa: move .qa terminator

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2022-02-03 16:40:07 -05:00
Cory Snyder
0713f65355 rgw: fix segfault in OpsLogRados::log when realm is reloaded
We weren't previously handling the deallocation of the store when
a realm was reloaded. Now passing a const reference to the pointer.

Fixes: https://tracker.ceph.com/issues/54130
Signed-off-by: Cory Snyder <csnyder@iland.com>
2022-02-03 21:31:17 +00:00
Andriy Tkachuk
9d1488f2b3 rgw_sal_motr: fix possible memleak on put
Currently, the MotrAtomicWriter::cleanup() is called from
MotrAtomicWriter::commit(), which may not be called at all
by rgw in case of md5 checksum failure.

Solution: call cleanup() from process() when data is zero.
rgw calls Writer::process(data, off) with zero data at the
end of the loop to allow writes to flush the data. From:
src/rgw/rgw_op.cc:RGWPutObj::execute():

      op_ret = filter->process(std::move(data), ofs);
      ...
      ofs += len;
    } while (len > 0);

    // flush any data in filters
    op_ret = filter->process({}, ofs);

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
Reviewed-by: Sining Wu <sining.wu@seagate.com>
2022-02-03 20:44:04 +00:00
Andriy Tkachuk
a9a2955297 rgw_sal_motr: fix cosmetic issues
- rename acc_bl to acc_data
- drop DEBUG prefix from logs

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
2022-02-03 20:42:17 +00:00
Kamoltat
f06da20dff pybind/mgr/progress: disable pg recovery event by default
The progress module disabled the pg recovery event by default
since the event is expensive and has interrupted other serviceis
when there is OSDs being marked in/out from the the cluster.

To turn the event on manually:

ceph config set mgr mgr/progress/allow_pg_recovery_event true

Updated qa/tasks/mgr/test_progress.py to enable
the pg recovery event when testing the progress module.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
2022-02-03 17:51:42 +00:00