Commit Graph

118921 Commits

Author SHA1 Message Date
Patrick Donnelly
f77e6aa5c8
test: add client tests
For alternate_name, so far.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:40 -08:00
Patrick Donnelly
e65b8dcad1
client: wire up alternate_name
Here we're exposing a public Client::walk (aka path_walk) so that the
user can inspect dentries (not something normally possible in POSIX).
We're going to skip exposing such an interface in libcephfs since
there's no reason to do that (who would use it?) except for testing.
Instead, a follow-up PR will add Client tests (for the first time, yay!)
that will exercise this code.

Also, ideally we'd also expose alternate_name via readdir results but
that is a bit more complicated since dirents do not normally refer to
external memory. So, just rely on Client::walk for testing for now.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:40 -08:00
Patrick Donnelly
b91490d353
mds: fix alternate_name durability
This is a collection of fixes to Xiubo's prior work. Namely:

- Add new mds_alternate_name_max option to limit the size of
  alternate_name. Otherwise a Client could trick the MDS into creating
  an alternate_name of any size!

- Clean up how alternate_name is assigned to CDentry. In the general
  case, this should be assigned as part of creating the dentry. We want
  this value to be immutable for the life of the dentry. Even for the
  very special case of rename(2) where the destination dentry already
  exists.  We explicitly check (after discussion with Jeff) that the
  target dentry alternate_name already matches what the rename RPC is
  giving.

- The MDS is now properly journaling the alternate_name.

- The MDS rejoin phase is properly transmitting each dentry's
  alternate_name. I've discovered that this MMDSCacheRejoin message
  actually wasn't versioned which I've raised in a tracker [1]. In the
  mean time, we'll just bump CEPH_MDS_PROTOCOL as usual.

[1] https://tracker.ceph.com/issues/48886

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:40 -08:00
Xiubo Li
9e250226a8
mds: add alternate_name feature support for dentries
This will support the "alternate_name" filename support, and will save
an alternate name for each dentry. This alternate name is not used in
path lookup or for any other usual file system purpose. The name is
simply an added blob of metadata on the dentry that is distributed to
clients so that "long" file names may be supported for clients which
require them. In the case of an fscrypt enhanced kernel mount driver,
the long name may be the cyphertext (exceeding FILENAME_MAX) of a long
file name.

Because this affects only files with long file names, the use of this
feature should be rare but could be common for some unusual
applications.

The client mount should check the CEPHFS_FEATURE_ALTERNATE_NAME feature
bit first to check whether the MDS has support for this feature or not.
The alternate_name is transmitted as part of the message payload in
MClientRequest when setting the alternate_name. The LeaseStat structure
in MClientReply contains the alternate_name.

When executing a metadata mutation RPC, the client will set the
alternate_name (if it exists) as part of the operation. The MDS will
pick that up and set it on the new or mutated dentry.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:39 -08:00
Xiubo Li
bd0b90ffbf
mds: add static encode/decode helpers for remote in CDentry
This will unify all the same work in different places.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-01-15 17:30:39 -08:00
Xiubo Li
7fe1c57846
mds: add "fscrypt" flag support for inode_t
If the client has set the "encryption.ctx" attribute, the MDS side
will set the "fscrypt" flag to truth, false as default.

The clients could use this flag to get to know whether the current
inodes are under enscrypted or not.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-01-15 17:30:39 -08:00
Xiubo Li
39f3440e36
mds: add new CDentry tags support
This will add new tag 'i' for inode and new tag 'l' for remote link
for the CDentry. And at the same time will add one proper dentry
version, which will be helpful to add new features/members in future
for the CDentry.

Fixes: https://tracker.ceph.com/issues/47162
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-01-15 17:30:39 -08:00
Patrick Donnelly
c73d012b5a
include: cleanup filepath cons
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:39 -08:00
Patrick Donnelly
805069fbe9
mds: add comment on linkage durability
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:39 -08:00
Patrick Donnelly
28b1c539c4
client: constify some RWRef methods
And make some Client state checks const.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:39 -08:00
Patrick Donnelly
093b0505e6
common: add strescape helper
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 17:30:38 -08:00
Neha Ojha
474a817421
Merge pull request #38730 from ronen-fr/wip-ronenf-scrub-interval
osd: improved handling of new-interval events by the scrubber

Reviewed-by: David Zafman <dzafman@redhat.com>
2021-01-15 17:02:19 -08:00
Neha Ojha
5c11f40c12
Merge pull request #38856 from dzafman/wip-48789
test: Fix osd-scrub-scaps.sh to handle DB format change

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-01-15 16:27:59 -08:00
Neha Ojha
6fc9166af4
Merge pull request #38726 from ronen-fr/wip-ronenf-48720
qa/standalone/scrub/osd-recovery-scrub: handle primary change when waiting for scrub

Reviewed-by: David Zafman <dzafman@redhat.com>
2021-01-15 13:46:30 -08:00
Patrick Donnelly
60510411be
Merge PR #38160 into master
* refs/pull/38160/head:
	doc/mgr/volumes: Document 'fs subvolume authorized_list' cli
	qa: Add tests for list auth-ids of a subvolume
	mgr/volumes: Update the 'volumes' key to 'subvolumes' in auth metadata file
	mgr/volumes: Optionally authorize existing auth-ids
	mgr/volumes: Preserve existing caps while authorize/deauthorize auth-id
	mgr/volumes: Disallow authorize existing auth_id
	mgr/volumes: Add subvolume authorized_list command
	mgr/volumes: Add tenant_id option to subvolume authorize
	mgr/volumes: Persist auth and subvolume metadata

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 11:53:06 -08:00
Mykola Golub
aab9de639f
Merge pull request #38862 from dillaman/wip-librbd-migration-qcow
librbd/migration: basic QCOW format handler

Reviewed-by: Mykola Golub <mgolub@suse.com>
2021-01-15 20:58:30 +02:00
David Zafman
af9befb0f4 test: Fix osd-scrub-scaps.sh to handle DB format change
Caused by: f9c95fa7fc

Fixes: https://tracker.ceph.com/issues/48789

Signed-off-by: David Zafman <dzafman@redhat.com>
2021-01-15 10:35:30 -08:00
Ernesto Puerta
7a503ca705
Merge pull request #38900 from Exotelis/update-i18n
mgr/dashboard: Update locale en to en-US to match transifex

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2021-01-15 19:07:16 +01:00
Kefu Chai
014e83728d
Merge pull request #38722 from tchaikov/wip-mgr-args-ann
mgr: use annotations for argdesc

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-01-16 01:51:27 +08:00
Kefu Chai
9d7248181a
Merge pull request #38899 from ronen-fr/wip-ronenf-blue-dangling
os/bluestore: fixing a dangling pointer

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-01-16 01:49:46 +08:00
Kefu Chai
ccc43feceb
Merge pull request #38902 from ifed01/wip-ifed-fix-deleting
osd/PG: fix no exit() call for PeeringState::Deleting state

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-01-16 01:48:30 +08:00
Kefu Chai
dc5d090edc
Merge pull request #38857 from dzafman/wip-48842
test: osd-recovery-prio.sh replace sleep with wait for both PGs recovering

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-01-16 01:46:43 +08:00
Kefu Chai
3fb4fc54ff
Merge pull request #38762 from peng-jiaqi/master
mgr: fix deadlock in ActivePyModules::get_osdmap()

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-01-16 01:44:38 +08:00
Kefu Chai
b1afb0ca6a
Merge pull request #38878 from tchaikov/wip-ceph-time-print
common/ceph_time: print {milli,nano}seconds as float

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2021-01-16 01:41:04 +08:00
Mark Nelson
f44d456621 common/options: Set pacific+ rocksdb column family options.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2021-01-15 12:30:39 -05:00
Adam Kupczyk
fe834c2100 BlueStore: Add block_cache logic to column family definition
Modified original onode column family cache into generic feature.
Now 2 options are possible for each column family:
1) use generic block cache but apply different BlockBasedTableOptions
2) create separate block cache. it will be applied for all shards of column family

It is done by specifying special option "block_cache" to CF definition:
Example: O(3,0-13)=block_cache={high_ratio=1.000}

"block_cache" accepts all BlockBasedTableOptions options with additions:
-"type" - binned_lru/lru/clock (default: ceph_options.rocksdb_cache_type)
-"size" - e.g.: 100M           (default: ceph_options.rocksdb_cache_size)
-"high_ratio" - e.g.: 0.75     (default: 0.0)
If any of above is set, new block cache is created, otherwise default is used.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2021-01-15 11:59:34 -05:00
Casey Bodley
b909583762
Merge pull request #38903 from yuvalif/wip-yuval-fix-48869
rgw/amqp: fix race condition in amqp manager initialization

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-01-15 11:23:26 -05:00
Mark Nelson
303d42533d BlueStore: Add onode column family KV block_cache.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2021-01-15 11:20:19 -05:00
Kefu Chai
ae7faa9db5 doc/mgr: document CLICommand and friends
the annotation and docstring are critical after we use them for
describing the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 23:11:58 +08:00
Thomas Goirand
201b592043 common/ipaddr: Allow binding on lo
Commmit 5cf0fa8722, solves the issue that
the osd can't restart after seting a virtual local loopback IP. However,
this commit also prevents a bgp-to-the-host over unumbered Ipv6
local-link is setup, where OSD typically are bound to the lo interface.

To solve this, this single char patch simply checks against "lo:" to
match only virtual interfaces instead of anything that starts with "lo".

Fixes: https://tracker.ceph.com/issues/48893
Signed-off-by: Thomas Goirand <zigo@debian.org>
2021-01-15 22:38:35 +08:00
Jason Dillaman
652963e7df librbd/migration: address code review comments
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2021-01-15 09:33:40 -05:00
Sebastian Wagner
be473e78ec
Merge pull request #38815 from sebastian-philipp/cephadm-osd-queue-race
mgr/cephadm: lock multithreaded access to OSDRemovalQueue

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-01-15 12:26:33 +01:00
Ernesto Puerta
865bbc3cf3
Merge pull request #38865 from s0nea/wip-dashboard-48847
mgr/dashboard: add `--ssl` to `ng serve`

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2021-01-15 12:07:07 +01:00
Sebastian Wagner
05f3b1d958
Merge pull request #38919 from tchaikov/wip-cephadm-yield_fixture
mgr/cephadm: s/yield_fixture/fixture/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-01-15 10:38:05 +01:00
Kefu Chai
b70f36bf91 pybind/mgr/mgr_module: annotate some methods
as a vehicle to understand the interface

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
42b41f897d mgr/dashboard/awsauth: drop py2 support
silences the warning from mypy:

dashboard/awsauth.py:42: error: Module 'base64' has no attribute
'encodestring'

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
620e318b2e doc/_ext: use iterator of tuple to build dict
As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
2c3e1926de mgr/dashboard: disable unsubscriptable-object warning from pylint
see https://github.com/PyCQA/pylint/issues/3882

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
0fb8a988df doc/_ext: PEP8 compliance
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
405f0e9b2d pybind/mgr/orchestrator: fix typo in comment
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
d25ed64449 doc/_ext: do not mock ceph_argparse
this module is used for convert argspec to argdesc by
CLICommand.load_func_metadata()

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
925c4d8d04 doc/_ext: add python_path option to ceph-mgr-commands
so we can add more python paths in addition to the one used for finding
mgr modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
fbbe4474ac doc/_ext: parse prefix and args for command signature
unlike the commands defined by C++, the commands defined by python
now uses "prefix" and "args" properties of elements in COMMAND class
attribute to define their command and arguments. the "cmd" property is
still available for the ceph CLI.

but the ceph_command sphinx extension should now use "prefix" and "args"
for printing out the usage of commands implemented using python. in
this change, the extension is updated to read "prefix" and "args"
properties of command defined by python modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
e6189427f3 pybind/mgr/orchestrator: do not pass cmd_args or desc to CLICommand anymore
they will be attached to the handler function as annotations and docstring

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:27 +08:00
Kefu Chai
3e41343db9 pybind/mgr/orchestrator: use enum parameter when appropriate
and move desc as docstring.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 17:25:06 +08:00
peng jiaqi
8e531ede1e mgr: fix deadlock in ActivePyModules::get_osdmap()
In function "ActivePyModules::get_osdmap()", We do not read or write to
object "ActivePyModules", so it is safe to delete lock
"ActivePyModules::lock", and it can avoid other thread waiting for lock
"ActivePyModules::lock"

 Fixes: https://tracker.ceph.com/issues/48852

Signed-off-by: peng jiaqi <peng.jiaqi@zte.com.cn>
2021-01-15 16:14:22 +08:00
Kotresh HR
9f9f8adc47 doc/mgr/volumes: Document 'fs subvolume authorized_list' cli
Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-01-15 10:10:43 +05:30
Kotresh HR
1b98e63e75 qa: Add tests for list auth-ids of a subvolume
Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-01-15 10:10:22 +05:30
xuxuehan xuxuehan
9f08e0ea8f crimson/osd: make PGRecovery::start_primary_recovery_ops take objs recovered by UrgentRecovery into account
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-01-15 12:04:33 +08:00
Kefu Chai
4902d7c545 pybind/mgr/mgr_module: cast string to enum when collecting kwargs
as the parameters of handlers are properly typed, they are expecting
enum parameter, let's cast string parameter to enum if the callee claims
that it wants an enum.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-15 09:50:53 +08:00