Commit Graph

119332 Commits

Author SHA1 Message Date
Igor Fedotov
ea0fc57ef5 os/bluestore: fix a bug causing unexpected Onode's unpinned state.
There could be a race for Onodes put() and get() methods:

put()(pinned, nref=3)
  int n = --nref; (nref = 2)
  if (n == 2) {
    ..
    std::lock_guard l(ocs->lock);
    ...
    pinned = pinned && nref > 2; (= false)
    ...                                     get()
    if (r) {                                ++nref; (=3)
      n = --nref; (nref = 2)                return;
    }
    ...
    return

As a result nref = 2, pinned = false which is wrong

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2021-02-02 12:25:00 +03:00
Kefu Chai
391cd0cd46
Merge pull request #39205 from rzarzynski/wip-crimson-configurable-alien-threads-num
crimson: make the number of alien threads configurable.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
2021-02-02 14:04:59 +08:00
Paul Cuzner
804f34e475 cephadm:updates following bootstrap code clean up
Patch to align to new(ish) bootstrap code changes

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-02-02 15:42:00 +13:00
Paul Cuzner
af00a33322 cephadm: updated doc text
The phrase "recommended"  has been removed.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-02-02 14:22:43 +13:00
Paul Cuzner
bbe4fd83c5 cephadm: unit test updates for cluster-network handling
Tests updated to handle comma separated subnet lists

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-02-02 14:22:43 +13:00
Paul Cuzner
abd9287db0 cephadm: updated handling of cluster_network parameter
Parameter now supports a comma separated list of subnets and
detects the ipv4 or ipv6 setting on the cluster network to enable
osds to bind correctly.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-02-02 14:22:43 +13:00
Paul Cuzner
509f171a08 cephadm: install doc updated to include cluster-network parameter
Install guide updated to include a description of the --cluster-network
parameter. The text also links to the complete definition for cluster-network
on the rados/configuration/network-config-ref page.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-02-02 14:22:43 +13:00
Paul Cuzner
3f352c8af4 cephadm: unit test updates for new subnet function
Adds tests to validate the check_subnet function will work as
expected with various inputs

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-02-02 14:22:42 +13:00
Paul Cuzner
c391ccb97f cephadm: Add cluster network parameter to bootstrap cmd
Adds a --cluster-network parameter to the bootstrap cmd to
set the internal cluster network.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-02-02 14:22:42 +13:00
Josh Durgin
caa58fce7a
Merge pull request #38956 from zdover23/2021_Jan_19_release_notes_grammar
doc/PendingReleaseNotes: grammar and wording

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-02-01 15:31:48 -08:00
Sage Weil
b0ff67b7ff mgr/cephadm/upgrade: tolerate pre-pacific upgrade state
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 17:20:14 -06:00
Sage Weil
604d2636d7 mgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades
For octopus -> pacific, as with other recent releases, we need to scale
down the MDS cluster(s) to a single daemon before upgrading.  (This is
because the MDS intra-cluster protocols aren't fully versioned.)

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 17:20:14 -06:00
Sage Weil
1e60a10370 mgr/cephadm: fix capitalization, level; drop elipses of log msgs
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 17:20:14 -06:00
Sage Weil
fe72946729 mgr/cephadm/upgrade: match against any repo_digest, not image_id
The image id can vary across hosts and (most notably) docker vs podman.
Instead, use the repo_digest as an image identifier.

Unfortunately, a single image may have multiple digests, even within the
same registry, so keep a list of the digests for the image we are
upgrading to, and ensure that each container has a digest that matches at
least one of them.

This allows upgrade to proceed in mixed docker+podman clusters.  However,
it does not yet address a cluster with mixed CPU architectures, because
the container image will have different digest(s) for each architecture
build.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 17:20:14 -06:00
Sage Weil
e9333972c1 cephadm: return repo_digests (plural) in pull/inspect output
When we inspect a (pulled) image, return all of the repo digests.  Update
the mgr/cephadm code accordingly to match.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 17:20:14 -06:00
Sage Weil
fafdb1e47d Merge PR #39149 into master
* refs/pull/39149/head:
	qa/workunits/cephtool/test.sh: tolerate some settling for osd info vs dump test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-02-01 16:25:01 -06:00
Sage Weil
9f61c94ae7 mgr/cephadm: include container_image_digests in inventory
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
0826c45e0c cephadm: include image_digests list in 'ls' output
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
6741082eb3 vstart.sh: only extract first container digest
Otherwise the container_image value includes the same hash twice, separated
by a newline, causing all sorts of confusion.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
43ac324c38 mgr/cephadm: move release -> major translation to helper
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
a3c9df9197 mgr/cephadm/upgrade: tolerate old upgrade_state.target_versoin
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
8abd45c4c9 mgr/cephadm/upgrade: set require-osd-release when done with OSDs
After we've upgraded all of the osds, complete the major version upgrade
for core RADOS by setting the OSD cluster min.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
ceac086143 mgr: add lookup_release_name(int) to mgr interface
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
5147d0f930 mgr/cephadm: verify container image version after we pull it
Apply the version checks to make sure this upgrade/downgrade is possible.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
dcaf514bbf mgr/cephadm: only save version portion of version string
We don't want to 'ceph version ' prefix or sha1 suffix.  This matches us
up with 'orch upgrade start'

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
d31bed7941 cephadm: fix 'inspect' and 'pull'
With podman 2.2.1 I would get

ERRO[0000] Error printing inspect output: template: all inspect:1: function "json" not defined

Removing the 'json' portion of the query resolves it.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
95dd015bce mgr/cephadm/upgrade: implement N-2 version checks on upgrade start
- Prevent major upgrades that span > 2 releases
- Prevent downgrades to an rc or dev release

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 16:23:02 -06:00
Sage Weil
7a1ce980d6 cpatch: python 3.6 -> 3.8
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 15:52:14 -06:00
Sage Weil
6df56bb04e cpatch: new default base image
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 15:52:14 -06:00
Samuel Just
a27b03940f crimson/os/seastore/segment_cleaner: tolerate mark_space_used on closed segment
A journal segment can be closed while outstanding writes on that
segment have not yet updated metadata.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 13:29:22 -08:00
Samuel Just
dfb1adaed6 crimson/os/seastore/journal: tolerate journal rolls during commit
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 13:29:22 -08:00
Samuel Just
41ae1c497b crimson/os/seastore: add basic pipeline phases to TransactionManager
We need to ensure that the metadata preperation and completions happen
in order.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 13:29:19 -08:00
Samuel Just
ad14ecedb6 crimson/common/operation: add OrderedConcurrentPhase
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
86348b48b3 src/crimson/common: decouple handle from stage
We're going to introduce stages with other properties.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
260c7c4af9 crimson/os/seastore: fix find_hole
We actually need to propogate eagain errors now, fix to use do_until.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
bcf4c89adf crimson/os/seastore: annotate all assert_all instances
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
568a332fba store-nbd: support concurrent operations
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
e43e5eadc5 crimson/os/seastore: validate node immediately prior to adding pin
Mostly, operating on an invalid extent is harmless by construction --
it'll be caught by the read set check in submit_transaction.  However,
prior to adding an extent to the lba pin set, we need to check that it's
really still live, which we can do by checking that it and its immediate
parent are still valid.  Mutating the target extent would have
invalidated it, and replacing the target extent would necessarily have
involved a mutation to the parent, which would have invalidated it.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
d244530f92 crimson/os/seastore: expand error type for reads to include eagain
There are cases where reads may detect an invalid cached extent and require
the user to restart the transaction.

As a side effect, clean up errorators to assert cases where construction
is invalid (invalid refcounts, enoent or invarg on node read, etc).  In
such cases, we may eventually want to introduce an internal_error error
type, but asserting is fine for now as we do not expect users to handle
these cases.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
ae9d15f670 crimson/common/errorator: add extend_ertr helper for merging errorators
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
ef24a8cb2d crimson/os/seastore/cache: put root in t.mutated_extents and avoid special behavior
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
d3e5937c08 crimson/common/operation.h: permit moving Handle objects
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:39 -08:00
Samuel Just
ccbc728ca9 test/crimson/seastore/test_transaction_manager: add tests for concurrent IO
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:36 -08:00
Samuel Just
c7eb6e4002 crimson/os/seastore/cached_extent: add asserts for debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-02-01 12:48:16 -08:00
Ernesto Puerta
27c32caa19
Merge pull request #36518 from yongseokoh/fix-dashboard
mgr/dashboard: Fix missing root path of each session for CephFS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2021-02-01 21:46:44 +01:00
Sage Weil
5cf1345981 mgr/MetricTypes: condition encoding on feature bits
Do not encode new metric|config types for older peers.

The combination of boost::optional and boost::variant make this super
awkward to condition on the features. :(

Fixes: https://tracker.ceph.com/issues/49069
Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 14:25:42 -06:00
Josh Durgin
25651e57b5
Merge pull request #38805 from kamoltat/wip-mgr-pg-autoscaler-improve-out-of-box-experience
pybind/mgr/pg_autoscaler: avoid scale-down until there is pressure

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-02-01 11:33:55 -08:00
Ilya Dryomov
d89613f52d qa/tasks/rbd: don't wait for krbd symlink to be created/removed
Commit 3754c665a1 (":qa/tasks/rbd: test qemu on top of rbd
encryption") broke dev_create() for krbd by messing up the "wait
for the symlink to be created by udev" loop.  The rbd tool has been
synchronizing with udev internally since 2014, so rather than fixing
let's just drop it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-02-01 17:05:47 +01:00
Ilya Dryomov
0357c0215b qa/tasks/rbd: dev_create() expects a properties dict
In order to add encryption_format property, commit 3754c665a1
(":qa/tasks/rbd: test qemu on top of rbd encryption") changed the
dev_create() contract.  It now expects a properties dict, but the
main task routine wasn't updated and still passes role_images.

On top of that, role_images itself got broken.  It is supposed
to be a mapping from role to a corresponding image name, not to
a dict with an image_name key.  This affected generic_mount().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-02-01 17:05:47 +01:00
Zac Dover
fa62671167 doc/PendingReleaseNotes: grammar and wording
This commit vets the PendingReleaseNotes for Octopus.
This might not be the only commit that updates the
release notes, but it is the first by me.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2021-02-02 01:37:59 +10:00