Commit Graph

119420 Commits

Author SHA1 Message Date
Sage Weil
6df56bb04e cpatch: new default base image
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 15:52:14 -06:00
Samuel Just
a27b03940f crimson/os/seastore/segment_cleaner: tolerate mark_space_used on closed segment
A journal segment can be closed while outstanding writes on that
segment have not yet updated metadata.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 13:29:22 -08:00
Samuel Just
dfb1adaed6 crimson/os/seastore/journal: tolerate journal rolls during commit
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 13:29:22 -08:00
Samuel Just
41ae1c497b crimson/os/seastore: add basic pipeline phases to TransactionManager
We need to ensure that the metadata preperation and completions happen
in order.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 13:29:19 -08:00
Samuel Just
ad14ecedb6 crimson/common/operation: add OrderedConcurrentPhase
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
86348b48b3 src/crimson/common: decouple handle from stage
We're going to introduce stages with other properties.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
260c7c4af9 crimson/os/seastore: fix find_hole
We actually need to propogate eagain errors now, fix to use do_until.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
bcf4c89adf crimson/os/seastore: annotate all assert_all instances
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
568a332fba store-nbd: support concurrent operations
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
e43e5eadc5 crimson/os/seastore: validate node immediately prior to adding pin
Mostly, operating on an invalid extent is harmless by construction --
it'll be caught by the read set check in submit_transaction.  However,
prior to adding an extent to the lba pin set, we need to check that it's
really still live, which we can do by checking that it and its immediate
parent are still valid.  Mutating the target extent would have
invalidated it, and replacing the target extent would necessarily have
involved a mutation to the parent, which would have invalidated it.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
d244530f92 crimson/os/seastore: expand error type for reads to include eagain
There are cases where reads may detect an invalid cached extent and require
the user to restart the transaction.

As a side effect, clean up errorators to assert cases where construction
is invalid (invalid refcounts, enoent or invarg on node read, etc).  In
such cases, we may eventually want to introduce an internal_error error
type, but asserting is fine for now as we do not expect users to handle
these cases.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
ae9d15f670 crimson/common/errorator: add extend_ertr helper for merging errorators
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
ef24a8cb2d crimson/os/seastore/cache: put root in t.mutated_extents and avoid special behavior
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
d3e5937c08 crimson/common/operation.h: permit moving Handle objects
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
ccbc728ca9 test/crimson/seastore/test_transaction_manager: add tests for concurrent IO
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:36 -08:00
Samuel Just
c7eb6e4002 crimson/os/seastore/cached_extent: add asserts for debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:16 -08:00
Ernesto Puerta
27c32caa19
Merge pull request #36518 from yongseokoh/fix-dashboard
mgr/dashboard: Fix missing root path of each session for CephFS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2021-02-01 21:46:44 +01:00
Sage Weil
5cf1345981 mgr/MetricTypes: condition encoding on feature bits
Do not encode new metric|config types for older peers.

The combination of boost::optional and boost::variant make this super
awkward to condition on the features. :(

Fixes: https://tracker.ceph.com/issues/49069
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 14:25:42 -06:00
Josh Durgin
25651e57b5
Merge pull request #38805 from kamoltat/wip-mgr-pg-autoscaler-improve-out-of-box-experience
pybind/mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-02-01 11:33:55 -08:00
Sage Weil
8429c7a4f2 qa/tasks/ceph_fuse: do not createfs
It's not clear to me if anything relies on this behavior or not.. but it
prevents us from using the ceph-fuse task with a cephadm-deployed
fs that doesn't described by the task roles.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 10:51:18 -06:00
Sage Weil
c2d53135fd qa/tasks/cephfs/fuse_mount: pass admin_socket path
For ceph.py this comes from the ceph.conf.template, but it's not there
for cephadm.  Instead of inflicting this on the config inside the
container, just pass it to ceph-fuse incantation here.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 10:50:33 -06:00
Sage Weil
964cd3e028 qa/suites/fs/cephadm/multivolume: add basic multivolume test
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 10:50:33 -06:00
Sage Weil
c8e1f4c2b5 qa/suites/rados: move thrash-old-clients out of cephadm
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 10:46:22 -06:00
Ilya Dryomov
d89613f52d qa/tasks/rbd: don't wait for krbd symlink to be created/removed
Commit 3754c665a11e (":qa/tasks/rbd: test qemu on top of rbd
encryption") broke dev_create() for krbd by messing up the "wait
for the symlink to be created by udev" loop.  The rbd tool has been
synchronizing with udev internally since 2014, so rather than fixing
let's just drop it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-02-01 17:05:47 +01:00
Ilya Dryomov
0357c0215b qa/tasks/rbd: dev_create() expects a properties dict
In order to add encryption_format property, commit 3754c665a1
(":qa/tasks/rbd: test qemu on top of rbd encryption") changed the
dev_create() contract.  It now expects a properties dict, but the
main task routine wasn't updated and still passes role_images.

On top of that, role_images itself got broken.  It is supposed
to be a mapping from role to a corresponding image name, not to
a dict with an image_name key.  This affected generic_mount().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-02-01 17:05:47 +01:00
Zac Dover
fa62671167 doc/PendingReleaseNotes: grammar and wording
This commit vets the PendingReleaseNotes for Octopus.
This might not be the only commit that updates the
release notes, but it is the first by me.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2021-02-02 01:37:59 +10:00
Zac Dover
ebea923ffc doc:dev remove workbench sections
This PR removes the following two sections:

1. Getting Ceph Workbench
2. Linking Ceph Workbench With Your OpenStack Tenant

Fixes: https://tracker.ceph.com/issues/48942
Signed-off-by: Zac Dover <zac.dover@gmail.com>
2021-02-02 01:01:25 +10:00
Radoslaw Zarzynski
387834bae9 crimson: make the number of alien threads configurable.
It's particularly important for reads to have large-enough
thread pool in `AlienStore` as they are synchronous; that
is, e.g. `BlueStore` blocks on IO when handling reads.

This commit introduces a configurable allowing operators
to increase the number from `1` which we were limited to
before the change.

Naming similarity with `osd_op_num_threads_per_shard` is
intentional.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-02-01 15:22:05 +01:00
Kamoltat
daeb6f6ac0 mgr/pg_autoscaler: avoid scale-down until there is pressure
The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.

Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools

Edited workunit script to reflect the change
of how pgs are calculated and distrubted.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
2021-02-01 13:05:54 +00:00
Sebastian Wagner
4203a7f93d
Merge pull request #39200 from mitsu-ko/mypatch
doc/mgr/orchestrator: Unify the content of command and yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-02-01 13:22:50 +01:00
Kefu Chai
65285f8f08
Merge pull request #39201 from tchaikov/wip-sphinx-openapi
doc: silence warnings from openapi sphinx extension

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2021-02-01 20:15:23 +08:00
Kefu Chai
8a01bad75f
Merge pull request #39194 from tchaikov/wip-dashboard-enum
pybind/mgr/dashboard: do not install enum34

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2021-02-01 19:11:32 +08:00
Mitsumasa KONDO
53639c2b01 doc/mgr/orchestrator: Unify the content of command and yaml
Signed-off-by: Mitsumasa KONDO <kondo.mitsumasa@gmail.com>
2021-02-01 18:13:12 +08:00
Kefu Chai
1debd98565 doc: silence warnings from openapi sphinx extension
it prints out

skipping non-JSON example generation.

if an API does not contains example while ":example:" is specified for
openapi extension, and if this API is not a GET request, openapi
extention complains.

see also
9dbae9c9a6/sphinxcontrib/openapi/openapi30.py (L191)

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-02-01 17:57:57 +08:00
Sebastian Wagner
639fac3564
Merge pull request #39198 from tchaikov/wip-mypy.ini
mypy.ini: fix the unresolve conflict

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-02-01 09:48:04 +01:00
zdover23
e9a74d6951
Merge pull request #39196 from badone/wip-doc-python3-api
doc/rados/api: Update code samples to python3

Reviewed-by: Zac Dover <zac.dover@gmail.com>
2021-02-01 17:18:35 +10:00
Kefu Chai
0c926d26cd mypy.ini: fix the unresolved conflict
it was introduced by a0930a6310, i failed
to resolve this conflict when resolving the merge confliction.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-02-01 15:11:56 +08:00
Brad Hubbard
0060cc52f9 doc/rados/api: Update code samples to python3
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2021-02-01 11:14:16 +10:00
Kefu Chai
d6c6669032
Merge pull request #39106 from mgfritch/cephadm-apt-get
cephadm: use `apt-get` for package install/update

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2021-01-31 17:23:10 +08:00
Kefu Chai
15a53419ca
Merge pull request #39131 from sebastian-philipp/rook-disable-untyped
mgr/rook: disallow_untyped_defs = True

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
2021-01-31 17:22:20 +08:00
Kefu Chai
9c652fb305 mgr/ActivePyModules.cc: use wrappers for acquiring/releasing GIL
this change is a follow-up of
0601b31a53, for couple reasons

- document the guideline for locking when working with python GIL
- add primitives to extract the patterns for acquiring/releasing
  GIL. so they can be reused.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-31 17:17:04 +08:00
Kefu Chai
9d0023ba90 pybind/mgr/dashboard: do not install enum34
enum was introduced by python 3.4, but we require python3.6 and up.
so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-31 12:04:23 +08:00
Sebastian Wagner
a3555a22c8 doc: add rook_client._helper to mock_imports
This is due to `mgr/rook` now also includes `._helper`

Co-authored-by Kefu Chai <kchai@redhat.com>

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-01-31 11:19:01 +08:00
Sebastian Wagner
a0930a6310 mgr/rook: disallow_untyped_defs = True
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-01-31 11:18:58 +08:00
Sebastian Wagner
ea51f6990c mgr/rook: Fix possibly infinite loop in _execute_blight_job
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-01-31 11:18:12 +08:00
Kefu Chai
79b91ea62d
Merge pull request #39143 from adk3798/ha-rgw-data-dir
cephadm: fix get_data_dir calls for ha-rgw daemons

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
2021-01-31 10:34:46 +08:00
Kefu Chai
383af836d7
Merge pull request #39081 from ifed01/wip-ifed-perpg-instrumentation
os/bluestore: be more verbose when fsck detects stray per-pg omaps

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-01-31 10:31:13 +08:00
Kefu Chai
78cdd1e5a2
Merge pull request #39095 from tchaikov/wip-cephadm-fsid
cephadm: add fsid if --name is not specified

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-01-31 10:29:59 +08:00
Kefu Chai
461b384aeb
Merge pull request #39056 from sebastian-philipp/alerts-percent-d
mgr/alerts: interval can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-01-31 10:28:54 +08:00
Kefu Chai
aca5ed1442
Merge pull request #38930 from vbashkirtsev/master
client/fuse_ll.cc: use uint64_t for fuse_ll_forget() nlookup argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-01-31 10:28:13 +08:00