Commit Graph

119070 Commits

Author SHA1 Message Date
Or Ozeri
8498da2aca qa/tasks/rbd: cleanup passphrase file
This commit adds a cleanup for the passphrase file used to create/mount encrypted rbd images.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
2021-01-26 17:02:37 +02:00
Jason Dillaman
ff0a062871
Merge pull request #39022 from petrutlucian94/rbd_fixes
Windows RBD fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:47:40 -05:00
Jason Dillaman
6d7acb1033
Merge pull request #38984 from petrutlucian94/dll
Windows DLL support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:47:26 -05:00
Jason Dillaman
eedfe2d000
Merge pull request #39073 from tchaikov/wip-rbd-bench
rbd/bench: include used headers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:47:06 -05:00
Jason Dillaman
6aba1c7bba
Merge pull request #39049 from lixiaoy1/fix_spec
spec: enable RWL/SSD caches

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:46:44 -05:00
Jason Dillaman
3c4e73ad4f
Merge pull request #39044 from orozery/librbd-crypto-close-bug
librbd: do not share crypto image layers with ancestors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:46:35 -05:00
Jason Dillaman
988a6ac8d3
Merge pull request #39006 from orozery/librbd-luks-cli-test
test/librbd: add luks encryption cli test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:46:20 -05:00
Jason Dillaman
55919f34c5
Merge pull request #39005 from orozery/librbd-plain64-iv-fix
librbd: use 512 sector size for crypto IV

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:45:54 -05:00
Ilya Dryomov
fcb0334156
Merge pull request #39064 from idryomov/wip-doc-osdtimeout
doc: don't mention osdtimeout option in mount.ceph man page

Reviewed-by: Jeff Layton <jlayton@redhat.com>
2021-01-26 13:37:10 +01:00
Ilya Dryomov
4cbd952916
Merge pull request #39060 from idryomov/wip-krbd-msgr2
krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 11:46:08 +01:00
Kefu Chai
af43300146 install-deps.sh: set codename when installing libzbd
this was an oversight in e7e3b86762

also use updated build with sha1 of
1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.

and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:45:09 +08:00
Kefu Chai
36dde8861a pybind/mgr/cephadm: return -ENOENT if self.ssh_user is None
there is chance that self.ssh_user is None when handling "cephadm
get-user" command, so return -ENOENT and error messages in that case.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
34e96c5539 pybind/mgr/dashboard: use _get_localized_key for composing store key
Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
3ca955c4d5 pybind/mgr/prometheus: add assert to appease mypy
self.get_osd_perf_counters() might return None, add assert() to assure
mypy that it does not return None here.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
2a5a50f3ba pybind/mgr/cephadm/serve: add assert to appease mypy
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
2fd0a03192 pybind/mgr/cephadm/module: assure self.ssh_user is not None
to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
cb3a50cd36 pybind/mgr/cephadm/inventory: cast variable to expected type
to silence mypy warnings like:

cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
cb2de05a39 tools/setup-virtualenv.sh: do not pass --use-feature=2020-resolver to pip
pip use the new resolver by default now. so no need to pass this option
to it anymore.

This reverts commit fa9e2bfd4b.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
00f5a80201 pybind/mgr/mgr_module: annotate variable type
to silence following mypy error:

cephadm/module.py:1155: error: Incompatible types in assignment
(expression has type "None", variable has type "str")

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
53415832cd pybind/mgr/cephadm: rename variables to avoid name reuse
to applease mypy, otherwise, for instance, "err" would be reused for
holding the return value of self.mgr.check_mon_command(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
eb6603f1b5 pybind/mgr: annotate OSDMap and friends
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
7b2dd7eba4 pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()
it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
d139166ea7 pybind/mgr: define HandleCommandResult using NamedTuple
simpler this way, so we don't need to define __new__

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
ab0243b319 pybind/mgr: use numeric values of logging level for index
less magic values in code this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
6c3d817435 mgr/cephadm: use dict instead of defaultdict to silence mypy warning
otherwise we have:

1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
5272748e17 mgr/rook: handle the case where given osd is not found
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:02:24 +08:00
Kefu Chai
2d20390b17 mgr/dashboard: pass str as the 2nd param of set_store()
MgrModule.set_store() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
6cdbe8f9e8 mgr/progress: pass a str as the value of option to set_module_option()
MgrModule.set_module_option() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
1268f36f2b mgr/progress: pass id to constructor of Event
to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:

progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
dee8474474 pybind/mgr/insights: be PEP8 compliant
to applease flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
352cd2d461 pybind/mgr/insights: use CLICommand to define command
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
3422ba1bfc pybind/mgr/iostat: minor cleanups
* do not pass typename and self to super()
* do not add parentheses around condition expression

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
4ffcccf6a4 pybind/mgr/iostat: use correct types
for better readability and for the sake of correctness.

* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
9bcd08c375 mgr/iostat: use CLICommand to define command
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
8da98e55b9 pybind/mgr/mgr_module: annotate more methods
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 18:01:59 +08:00
Kefu Chai
30070be248 cephadm: refactor call() using asyncio.asyncio.StreamReader
simpler this way, also fix a couple issues:

* create a child watcher explicitly, see
  https://bugs.python.org/issue35621
* use StringIO for collecting outputs for better performance,
  instead of appending the lines to an existing str
* catch ValueError when reading from the stream reader,
  because StreamReader.readline() could raise ValueError when
  it reaches the buffer limit while looking for a separator.
  in this case, we should try again, in hope that the spawned
  process can feed the reader with more data which contains
  the separator (i.e., b'\n').
* backport ThreadedChildWatcher from Python 3.8 so we can
  run create_subprocess_exec() in non-main threads.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 17:48:32 +08:00
Kefu Chai
05aef6d660
Merge pull request #38929 from jdurgin/wip-cot-log
qa/tasks/ceph_manager: let c-o-t log errors

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-01-26 17:33:22 +08:00
Kefu Chai
cd3ee7be75
Merge pull request #38931 from trociny/wip-48821
osd: fix potential null pointer dereference when sending ping

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-01-26 17:28:30 +08:00
Kefu Chai
9e2b4eb25f
Merge pull request #38901 from gmayyyha/io_uring-48872
blk/kernel: fix io_uring got (4) Interrupted system call

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-01-26 17:25:41 +08:00
Kefu Chai
b07a4c684f
Merge pull request #38932 from tchaikov/wip-48905
mgr: update mon metadata when monmap is updated

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-01-26 17:23:19 +08:00
lixiaoy1
12ca24fc26 spec: enable RWL/SSD caches
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
2021-01-26 04:21:30 -05:00
Lucian Petrut
5c008725db cmake,win32*.sh: Windows DLL support
At the moment, the Windows binaries can only be linked statically.
This is less than ideal since:

* the resulting binaries can be quite large, especially when
including debug symbols
* librados and librbd cannot be used directly by 3rd party apps
* the build duration is increased

In order to do a dynamic build, we'll:

* add an option to win32_build.sh
* fix link order
* dynamically link boost
* disable the "-pie" flag when using Mingw, which generates incorrect
  executable entry points
* by default, cmake generates import libs for executables. The issue
  is that for rados.exe, it generates librados.dll.a, thus overriding
  the librados.dll import library, which breaks the build process.
  We'll configure cmake to skip generating import libs for executables.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2021-01-26 08:41:51 +00:00
Kefu Chai
f16ac13c13 rbd/bench: include used headers
should help with fixing following failure:

src/tools/rbd/action/Bench.cc:20:26: error: aggregate 'std::atomic<bool>
terminating' has incomplete type and cannot be defined

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-01-26 16:09:28 +08:00
Jason Dillaman
251990a146 librbd/migration: allow retrieving QCOW L2 table from L2 cache
This will be used in the next commit where the list-snaps state
machine is tweaked to compate L2 tables between snapshot revisions
to determine deltas.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 00:21:51 -05:00
Jason Dillaman
8368724402 librbd/migration: refactor lookup table handling for QCOW format
This will be used by a refactored list-snaps state machine to compute
the L2 table cluster for image offsets. The L1 and L2 lookup tables
can now be represented in a single structure.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 00:21:51 -05:00
Matt Benjamin
d3a78fa196
Merge pull request #37250 from cbodley/wip-qa-librgw
qa/rgw: add librgw_file unit tests to rgw/verify task
2021-01-25 22:10:37 -05:00
Jason Dillaman
7090ca4a08 librbd/migration: tweaks to QCOWFormat discovered during unit testing
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2021-01-25 20:16:04 -05:00
Harish Munjulur
46dee40a29
Merge pull request #38852 from adamemerson/wip-thou-shalt-not-unwind
Thou Shalt Not Unwind
2021-01-25 15:22:38 -08:00
Harish Munjulur
d358f62397
Merge pull request #38905 from cbodley/wip-48874
rgw: fix trailing null in object names of multipart reuploads
2021-01-25 15:17:05 -08:00
Neha Ojha
6533027912
Merge pull request #39063 from yuriw/wip-yuriw-crontab-master
qa/tests: added pacific branch to the mix

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-01-25 14:11:53 -08:00