Commit Graph

110914 Commits

Author SHA1 Message Date
Bryan Stillwell
82699067b8 compressor: Add a config option to specify Zstd compression level
Add a new configuration item called 'compressor_zstd_level' so that the
Zstandard compression level can be tuned to the workload on a cluster.

Fixes: https://tracker.ceph.com/issues/43377

Signed-off-by: Bryan Stillwell <bstillwell@godaddy.com>
2020-04-20 13:26:54 -06:00
Casey Bodley
6a64fe3e1a rgw: use target bucket layout for reshard
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-20 15:23:14 -04:00
Neha Ojha
4f82ebf41b qa/standalone/scrub/osd-scrub-repair.sh: fix race in TEST_auto_repair_bluestore_failed
We need to flush_pg_stats before checking for active+clean.

Fixed: https://tracker.ceph.com/issues/45075
Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-04-20 18:29:51 +00:00
Yuri Weinstein
158518fdcf qa/tests: initial checkin for clients upgrades tests for future release of pacific
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2020-04-20 10:58:57 -07:00
Jason Dillaman
73e3f537ef
Merge pull request #34608 from trociny/wip-journal-append-stripe-width
cls/journal: use EC pool stripe width for padding appends

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-04-20 13:27:35 -04:00
Kefu Chai
0a183082e5 mgr: drop python2 support
just stop checking for `PY_MAJOR_VERSION`, since we've moved to python3.
no need to be python2 compatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-20 23:39:20 +08:00
Casey Bodley
58a773b59c qa/rgw: allow 'ceph' user to read vault token
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-20 11:36:51 -04:00
Sebastian Wagner
67a29846f8 mgr/orch: upgrade: better input validation
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-20 15:45:02 +02:00
Kefu Chai
94e39db4e8
Merge pull request #34636 from tchaikov/wip-vstart-runner-py3
qa/tasks/vstart_runner,doc: be compatible with python3

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2020-04-20 21:23:43 +08:00
Dimitri Savineau
252d4b3044 cephadm: fix adopt prometheus command
Create the target directory before copy the prometheus configuration.

Closes: https://tracker.ceph.com/issues/45120

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-04-20 08:42:25 -04:00
Kefu Chai
c5bd318de6 qa/tasks/cephfs/fuse_mount.py: by python3 compatible
pass `StringIO()` to capture stdout whose value will be interpreted as a
string later on

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-20 19:55:09 +08:00
Zac Dover
c4765afb81 Add troubleshooting guidance for ssh connection failures.
For more thorough information about this commit, see:
https://tracker.ceph.com/issues/44905

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Michael Fritch <mfritch@suse.com>

ibid

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>

Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>

Added mgfritch's suggestion regarding checking to see if the public key
is listed in the authorized_keys file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2020-04-20 21:48:22 +10:00
Kefu Chai
d54f3ea43e pybind/mgr/dashboard/run-backend-api-tests.sh: use python3 by default
since vstart_runner.py is now py3, let's use python3 by default

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-20 19:41:09 +08:00
Kefu Chai
bedea85019 qa/tasks/vstart_runner.py: be python3 compatible
differentiate `str` and `bytes` instances, and drop python2 support from
vstart_runner.py, as we've moved to python3 already

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-20 19:41:09 +08:00
Igor Fedotov
0f639740a3 qa/objectstore: test hybrid allocator instead of avl one.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-04-20 14:33:14 +03:00
Igor Fedotov
c6b2413e8e os/bluestore: fix unittest_bluefs.mkfs_mount_duplicate_gift
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-04-20 14:06:48 +03:00
Xiubo Li
3441261346 qa/volume_client: switch to use python3
Currently for the ceph_volume_client in pybind only the python3 is
support.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-20 07:05:03 -04:00
Xiubo Li
0ed507dfd4 qa/volume_client: fix TestVolumeClient failing under new py3 runtime
The pybind now has dropped the WITH_PYTHON2 option, and for now only
py3 supported.

Fixes: https://tracker.ceph.com/issues/45103
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-20 07:04:52 -04:00
Kefu Chai
a2d1eaeb3f qa/suites/*/rados_cls_all.yaml: load all classes
the intention to add the whitelist was to test "sdk" class, but if we
add new classes to the list, and add tests exercising them, the tests
fail if we fail to update these `rados_cls_all.yaml` accordingly.

so in this change, the list is now '*' which allows OSD to load all
classes found in the specified directory

Fixes: https://tracker.ceph.com/issues/45113
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-20 18:58:35 +08:00
Kefu Chai
c6e48ef0f0 doc/dev/developer_guide: use python3 to launch vstart_runner.py
as vstart_runner.py is compatible with python3, and so is the tests
under `qa/` directory.

and use `pip` to install teuthology, simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-20 18:31:37 +08:00
Yan, Zheng
f558dff017 mds: make MDCache::scan_stray_dir() handle dir fragmentation
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-04-20 18:11:18 +08:00
Yan, Zheng
9a308ed4a4 mds: don't add null dentry to frozen stray dir
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-04-20 18:11:18 +08:00
Yan, Zheng
72dab35bd0 mds: auth pin stray dir when purging stray inode
After stray inode gets added to purge queue, mds deletes stray dentry.
MDS needs to make sure stray dir is not being fragmented.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-04-20 18:11:15 +08:00
Yan, Zheng
02e15a20d9 ceph: allow 'dirfrag split' asok command to operate on stray dirs
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-04-20 18:06:26 +08:00
Kefu Chai
36ddb9ed3b
Merge pull request #34432 from optimistyzy/47_upstream
os/bluestore/NVMEDevice.cc: use slist for tracking mempool

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-04-20 14:51:33 +08:00
Ziye Yang
30858a9534 NVMEDevice: Remove the performance counter
Compared with other approach (KernelDevice, iouring), it does not
have such feature, so remove it to simplify the code and improve
the performance. Because, each thread will call the logger and calculate
the time, it is very time consuming. And the upper layer in bluestore has
the feature to track each I/O task.

And it is not readable if we add ifdef configure to enable this feature,
so in my mind that I suggest removing this latency track. And the
latency benefit demonstrated by SPDK is already approved, so we do not
need to tracking every I/O time when invoking SPDK driver.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2020-04-21 00:25:08 +08:00
Ziye Yang
bb55f2c2a3 NVMEDevice.cc: Remove the data_buf_mempool
Use the linked list is enough.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2020-04-21 00:24:38 +08:00
Yuan Lu
bee4e6f66c librbd: add aio_flush test case
Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>
2020-04-20 14:04:05 +08:00
Yuan Lu
03f08687bb librbd: add aio_flush
Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>
2020-04-20 14:03:45 +08:00
Mykola Golub
5e540f1191
Merge pull request #34615 from dillaman/wip-45072
rbd-mirror: image replayer stop might race with instance replayer shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>
2020-04-19 10:57:49 +03:00
Yuval Lifshitz
73db460e52 rgw/amqp: fix the "routable" delivery mode
this option was not exposed to the configuration API
however, it was still set, as hardcoded value in the code
(details:
https://www.rabbitmq.com/confirms.html#publisher-confirms)

Fixes: https://tracker.ceph.com/issues/44915

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2020-04-19 10:25:56 +03:00
Xiubo Li
cb0a600acf ceph-fuse: link to libfuse3 if fuse3 is installed
Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-18 05:57:29 -04:00
Xiubo Li
32a45b624b rbd-fuse: remove FUSE_CAP_BIG_WRITES when linking to libfuse3
The big-writes is enabled as default from 3.0.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-18 05:50:27 -04:00
Xiubo Li
d5999a9ac4 rbd-fuse: switch to use utimens instead
The utime is deprecated.

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-18 05:50:22 -04:00
Kefu Chai
c4a0acd91d
Merge pull request #34576 from jdurgin/wip-object-corpus
ceph-object-corpus: update to octopus

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-04-18 11:59:55 +08:00
Kefu Chai
b557467df0
Merge pull request #34535 from changchengx/refine_build_liburing
cmake: refactor liburing support

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-04-18 11:39:32 +08:00
Xiubo Li
1c3dd60f21 ceph-fuse: enable the big writes as default
When the libfuse version < 3, without the big_writes options enabled
the write perf is very poor.

Fixes: https://tracker.ceph.com/issues/44885
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-17 20:40:18 -04:00
Igor Fedotov
cacc7391dc os/bluestore: set hybrid allocator as a default for bluefs/bluestore
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-04-18 02:45:19 +03:00
Sage Weil
7eb29b3cd5 cstart.sh: work with docker or podman
Signed-off-by: Sage Weil <sage@redhat.com>
2020-04-17 18:05:44 -04:00
Josh Durgin
e94b553816 src/script/cpatch: complete globs in correct paths
Otherwise none of these libs are copied

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2020-04-17 18:05:44 -04:00
Josh Durgin
0d5c6b6cc0 src/script/cpatch: include cython libs and cephadm
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2020-04-17 18:05:44 -04:00
Sage Weil
be876835c1 src/cstart.sh: initial version
Signed-off-by: Sage Weil <sage@redhat.com>
2020-04-17 18:05:44 -04:00
Sage Weil
6f9e51ca1a script/cpatch: initial version
Based on kubejacker.

- copy + strip (tar compress is sloooow)
- avoid touching build artifacts
- by default, patch "everything"
- optionally do actual build as root
- lump rbd and cephfs into core because mgr needs to dynamically link
  them and c++ ABI is fragile.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-04-17 18:05:44 -04:00
Michael Fritch
b06f5b7850
qa/workunits/cephadm/test_cephadm.sh: extend timeout test
test occasionally fails when the container is slow to start

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-04-17 15:01:55 -06:00
Michael Fritch
879e685099
test_cephadm.sh: remove unmaintained test_cephadm.sh
the script has become unmaintained and duplicates logic found in
qa/workunits/cephadm/test_cephadm.sh

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-04-17 14:26:29 -06:00
Michael Fritch
6b21144723
qa/workunits/cephadm/test_cephadm.sh: add CLEANUP var
preserve the current test state during EXIT:
`CLEANUP=false ../qa/workunits/cephadm/test_cephadm.sh`

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-04-17 14:17:02 -06:00
Sebastian Wagner
138018eddc
Merge pull request #34584 from bk201/wip-45108
mgr/test_orchestrator: fix error when listing services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-17 18:55:21 +02:00
Sebastian Wagner
fe91dfad74
Merge pull request #32951 from matthewoliver/cephadm_rgw_ssl
cephadm: add RGW SSL support

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-17 18:54:50 +02:00
Gregory Farnum
c17f4d9262
Merge pull request #34410 from lxbsz/mds_dirty_parent
tools/cephfs: add accounted_rstat/rstat when building file dentry

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>
2020-04-17 08:27:40 -07:00
Gregory Farnum
62dadb2a33
Merge pull request #33928 from mchangir/cephfs-journal-tool-cannot-set-dry_run-arg
cephfs-journal-tool: correctly parse --dry_run argument

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2020-04-17 08:26:52 -07:00