Commit Graph

117806 Commits

Author SHA1 Message Date
Jason Dillaman
be6fd6eb3a
Merge pull request #38389 from trociny/wip-48412
librbd: fix regression in object map diff request

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>
2020-12-05 15:37:51 -05:00
Kefu Chai
2272f4b658
Merge pull request #35821 from Aran85/fix-ec-read
osd/ECBackend: optimize remaining read as readop contain multiple objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
2020-12-05 22:54:33 +08:00
Matt Benjamin
9264e6a32f
Merge pull request #37543 from ianwatsonrh/master
docs: With the addition of STS x-amz-security-token is now supported
2020-12-05 09:46:09 -05:00
Kefu Chai
8f028cd9c6
Merge pull request #38401 from a16bitsysop/alpine-tests
test/lazy-omap-stats: Fix compilation on alpine linux

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:12:48 +08:00
Kefu Chai
670efdd3c7 doc: build api docs with Read the Docs
since we are able to build the python bindings using the stub
implementation of C binding, let's enable the API docs build.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:22 +08:00
Kefu Chai
86d9df5a3c pybind: check for READTHEDOC env variable
for building stub C binding if building in Read the Docs environment,
see https://docs.readthedocs.io/en/stable/builds.html

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:22 +08:00
Kefu Chai
2235c2db0e admin/build-doc: do not build stub C binding anymore
since the python binding is able to build the stub by themselves, there
is no need to do this manually.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:22 +08:00
Kefu Chai
17ce94e71e pybind/cephfs: use stub implementations when building document
also define a stub for rados.Rados right in cephfs.pyx, so the cython
compiler needs it. but it's cubersome to copy rados.pxd from rados
python binding when building cephfs python binding. because, if we
install a local python package using requirement.txt, pip does not allow
us to specify the include-directory for Cython. and cephfs/setup.py is
located out of the source tree when it is being built by pip, and unlike
in our CMake based build, which specifies --cython-include-dirs with
absolute directory, we don't have access to the root directory of
project in Read the Docs environment, so it's impossible for us to
locate pybind/rados in setup.py.

we could pass "--global-option" as part of package specifier in
requirements.txt, like:

cephfs --global-option="build_ext" --global-option="--for-doc"

or

cephfs --global-option="build_doc"

but then we will have to override or create a command
for interpreting the command line options.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:22 +08:00
Kefu Chai
12140f11d0 pybind/cephfs: drop duplicated declaration
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:22 +08:00
Kefu Chai
062e2bed9c pybind/cephfs: extract non-shared interface definitions
* c_cephfs.pxd: for cephfs C declarations
* types.pxd: for types exposed by libc and kernel

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:22 +08:00
Kefu Chai
ccc247d1b0 pybind/rbd: use stub implementations when building document
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:22 +08:00
Kefu Chai
e63d4a8fe0 pybind/rbd: extract non-shared interface definitions into c_rbd.pxd
so it'd be simpler if we want to parse and generate the stub
for these declarations when building document where where the
annotations are used.

ctime.pxd is extracted so the time related declarations can be shared by
c_rbd.pxd and rbd.pyx.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
778fa7185d pybind/rgw: use stub implementations when building document
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
a05f624288 pybind/rgw: s/bool/bint/
when compile a pure python cythonized binding, bool is not defined
because, i think, Cython does not include <stdbool.h> by default.
so use bint indead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
ea80b9b40e pybind/rgw: extract non-shared interface definitions into c_rgw.pxd
so it'd be simpler if we want to parse and generate the stub
for these declarations when building document where where the
annotations are used.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
98ab937a3a pybind/rados: use stub implementations when building document
* setup.py: inject "BUILD_DOC" into cython build context if "BUILD_DOC"
    env variable is set, this helps us to do conditional compilation.
    for instance, if we are building document, there is no need to
    use the header files or link against librados.so for building
    the python bindings.
* mock_rados.pxi: implement the mock functinons for their C counterparts
    in librados, so we can use them when building librados python
    binding for creating the document. Sphinx's autodoc extension
    reads annotation of the python package whose document is being
    built, after importing the python package into python runtime.
    but since we use Cython for creating our python binding,
    we have to provide the implementation of those referenced
    librados C API symbols. mock_rados.pxi implements them using Cython.
    the downside of this solution is that we need to mirror every
    function used by the Python binding by repeating them in both
    c_rados.pxd and mock_rados.pxi. the same will apply to other
    Ceph python bindings as long as they use C APIs not offered by
    libc or python runtime. so next step is to develop a simple
    parser which can be run at build time to create mock_*.pxi from
    c_*.pxd.
* rados.pyx: use mock_rados.pxi if BUILD_DOC, otherwise use
    c_rados.pyd. the latter requires a librados.so at runtime.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
645a05067d pybind/rados: check if BUILD_DOC in os.environ
no need to check if it is in keys()

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
2580aa2141 pybind/rados: remove unused import statement
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
b86ed55535 pybind/rados: extract non-shared interface definitions into c_rados.pxd
so it'd be simpler if we want to parse and generate the stub
for these declarations when building document where where the annotations
are used, but importable python bindgs of rados,ceph,rbd,cephfs are required.

also move the time.h declarations into ctime.pxd, are they can be shared
by the stub implementation and the non-stub one.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-12-05 22:09:21 +08:00
Kefu Chai
fb825ddd2f
Merge pull request #38314 from badone/wip-monitor-reset-abort-if-is_shutdown
mon: paxos: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-12-05 21:59:02 +08:00
Kefu Chai
c165fc82dc
Merge pull request #38410 from badone/wip-monitor-logger-deelete-in-destructor
mon: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-12-05 21:53:58 +08:00
Kefu Chai
4ae8abbe21
Merge pull request #37314 from ifed01/wip-ifed-faster-rm-p1
osd: optimize PG removal (part1)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
2020-12-05 21:50:30 +08:00
Kefu Chai
f634441a9c
Merge pull request #38438 from ifed01/wip-ifed-fix-avl
os/bluestore: fix inappropriate ENOSPC from avl/hybrid allocator

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-12-05 21:49:11 +08:00
Kefu Chai
9551d8b354
Merge pull request #38436 from ktdreyer/prometheus-err
mgr/prometheus: don't store exception as e

Reviewed-by: Boris Ranto <branto@redhat.com>
2020-12-05 21:38:54 +08:00
Brad Hubbard
593a8e8e98 cmake: require fmt 6.0.0 and up
The cmake search for a local package has lagged in regard to our usage
of it.

Fixes: https://tracker.ceph.com/issues/48453

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2020-12-05 17:56:38 +10:00
Kefu Chai
33440ee973
Merge pull request #38400 from tchaikov/wip-pybind-use-non-deprecated-functions
pybind/rados: use rados_set_pool_full_try instead of the deprecated one

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-12-05 12:14:28 +08:00
Boris Ranto
d98d141c4f mgr/crash: Serialize command handling
All the implemented commands read or write the self.crashes structure.
We need to serialize them to avoid the threads from stepping over each
other toes.

This also makes sure that the main thread (serve method) does not
interfere with the commands.

Signed-off-by: Boris Ranto <branto@redhat.com>
2020-12-05 04:14:35 +01:00
Jeff Layton
4181742a3b qa: allow arbitrary mount options on kclient mounts
With this, we can pass a list of mount options to the kclient that will
be added onto the default ones. This is necessary to support testing
with fscache enabled (and other features activated by mount options).

Fixes: https://tracker.ceph.com/issues/6373
Signed-off-by: Jeff Layton <jlayton@redhat.com>
2020-12-04 18:22:28 -05:00
Matt Benjamin
3dde92561f
Merge pull request #36384 from linuxbox2/wip-doc-ippolicy
rgw:doc: document support for (Not)IpAddress in S3 policy
2020-12-04 17:37:58 -05:00
Igor Fedotov
d4011fd1fb os/bluestore: bring back probing available space for main dev allocator.
This feature was mistakenly removed while making a single allocator
for main device (https://github.com/ceph/ceph/pull/30838).

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-12-05 00:45:06 +03:00
Samuel Just
2bf1388228
Merge pull request #37546 from myoungwon/wip-revamp-tier-flush
osd: refactoring tier_flush()

Reviewed-by: Samuel Just <sjust@redhat.com>
2020-12-04 13:39:41 -08:00
Igor Fedotov
ba9b547120 doc/osd-config-ref: adjust default value for osd_delete_sleep_hybrid
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-12-05 00:09:52 +03:00
Igor Fedotov
6e09f0c13a osd: adjust osd_delete_sleep_ssd[_hybrid]
1) Do not differentiate all-flash and hybrid(ssd+hdd) deployments since
they both bound to SSD drive backing DB volume.
2) Drop osd_delete_sleep_hybrid from 2s to 1s
3) Raise osd_delete_sleep_ssd to 1s since 0 is a way too low - having
it that low tend to cause a significant negative imact on OSD's regular
operation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-12-05 00:09:52 +03:00
Igor Fedotov
cc9d58421e os/bluestore: introduce removal perf counter
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-12-05 00:09:52 +03:00
Igor Fedotov
0504b95549 os/bluestore: track kv utilization in kv_sync_thread
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-12-05 00:09:52 +03:00
Igor Fedotov
86840616ba os/bluestore: perf counters for omap_get_keys[_get_values]
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-12-05 00:09:52 +03:00
Igor Fedotov
7f047005fc osd/pg: use next when calling collection_list for pg removal
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-12-05 00:09:50 +03:00
Jason Dillaman
4d86fc49ea
Merge pull request #38449 from trociny/wip-rbd-flush-finisher
librbd: flush task finisher queue on unregistering watcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-12-04 14:50:01 -05:00
Jason Dillaman
24d2005fa2
Merge pull request #38399 from es-gyt/master
rbd: current pc value should be greater than last_pc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-12-04 14:00:43 -05:00
zdover23
650d35ee22
Merge pull request #38403 from zdover23/wip-doc-CLT-update-02-Dec-2020-1
doc/governance.rst - updating CLT list

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-12-05 03:49:48 +10:00
Ernesto Puerta
37ade59f24
Merge pull request #38440 from bk201/wip-48454
mgr/dashboard: fix trusted_ip_list in iSCSI service form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-12-04 18:40:03 +01:00
Kefu Chai
e940450544
Merge pull request #38398 from tchaikov/wip-48210
pybind/mgr: unpin urllib3

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-12-05 01:33:24 +08:00
Ernesto Puerta
4ae467e979
Merge pull request #38298 from bk201/wip-43165
mgr/dashboard: refactor /api/orchestrator/* endpoints

Reviewed-by: Courtney Caldwell <ccaldwel@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2020-12-04 18:24:46 +01:00
Ernesto Puerta
b0c33a4bc4
Merge pull request #38432 from rhcs-dashboard/orch-not-available-incorrect-status
mgr/dashboard: Fix for misleading "Orchestrator is not available" error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-12-04 18:22:43 +01:00
Mykola Golub
3f8c63175d test/librbd: fix race in TestLibRBD.ConcurentOperations
It was possible that before image1 was closed, both quiesce2 and
quiesce3 requests were received and the test got stuck on
create_snap1.join() waiting for the image1 to be closed, while
the close was waiting for quiesce3 to be completed.

Signed-off-by: Mykola Golub <mgolub@suse.com>
2020-12-04 17:02:17 +00:00
es-gyt
8e36435d62 rbd: current pc value should be greater than last_pc
If rbd image enabled striping feature, stderr will
print repeat rate of progress when execu rbd export-diff.

Fixes: https://tracker.ceph.com/issues/48431
Signed-off-by: Yite Gu <yite.gu@easystack.cn>
2020-12-05 00:01:35 +08:00
Mykola Golub
4939841633
Merge pull request #38379 from dillaman/wip-librbd-migration-5
librbd/migration: add thick-provisioned snapshot support to the raw format

Reviewed-by: Mykola Golub <mgolub@suse.com>
2020-12-04 17:43:59 +02:00
Sebastian Wagner
7315f4c487
Merge pull request #38285 from jmolmo/include_hw_info_in_HostCache
mgr/cephadm: Get host facts data in host cache

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-12-04 16:11:08 +01:00
Volker Theile
9d4282e53a mgr/dashboard: test_standby* (tasks.mgr.test_dashboard.TestDashboard) failed locally
Fixes: https://tracker.ceph.com/issues/48449

Signed-off-by: Volker Theile <vtheile@suse.com>
2020-12-04 14:39:37 +01:00
Zac Dover
d7f88e3ce4 doc/governance.rst - updating CLT list
This PR removes the following people from the
CLT. They've left the CLT:
* Lenz Grimmer
* Alfredo Deza
* Zack Cerza

This commit also adds:
* Ernesto Puerta

Fixes: https://tracker.ceph.com/issues/48436
Signed-off-by: Zac Dover <zac.dover@gmail.com>
2020-12-04 23:31:45 +10:00