Commit Graph

860 Commits

Author SHA1 Message Date
RAJAT SINGH
5c49aab11e rbd: add GetMirrorUUID function
* Add GetMirrorUUID implementing rbd_mirror_uuid_get

This function is be used to get the mirroring uuid for the pool.
Basic tests included.

Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
2021-06-08 14:01:09 +00:00
John Mulligan
24eeb9512c rbd: add a MirrorImageGlobalStatusIter implementing rbd_mirror_image_global_status_list
This adds the function call in the style of an iterator, as the number
of mirrored images in the entire pool could be large. Tests are
included.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-08 12:20:53 +00:00
Sébastien Han
8179bd4437 rgw/admin: stop returning pointers
Since we are not passing a pointer, let's not return a pointer either.
Also, our use case fits well the non-returning pointer approach since
the content of the struct type is not expected to be modified.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
Sébastien Han
a2c50e28a5 rgw/admin: add quota support
This commit introduces support for user and bucket quota on the rgw
admin ops API.

Co-authored-by: Irek Fasikhov malmyzh@gmail.com
Co-authored-by: Quentin Perez qperez42@gmail.com
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
Sébastien Han
7b585cc9b4 rgw/admin: add new error for SignatureDoesNotMatch
It indicates a mal-formed query for the API where some parameters are
incorrect. This is unlikely to happen with good integration test but we
never know.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
Sébastien Han
cf09833810 rgw/admin: remove unnecessary url.Parse() call
http.NewRequestWithContext() already calls url.Parse() internally so we
don't need call it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
John Mulligan
2be2128e33 rbd: add CreateMirrorPeerBootstrapToken & ImportMirrorPeerBootstrapToken functions
* Add CreateMirrorPeerBootstrapToken implementing rbd_mirror_peer_bootstrap_create
* Add ImportMirrorPeerBootstrapToken implementing rbd_mirror_peer_bootstrap_import

These functions can be used to set up mirroring between pools. Basic tests
included. The tests only verify that functions work, not that they
actually mirror data. That is a job for another day.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-03 14:42:42 +00:00
Mudit Agarwal
444050570c rados: add SetPoolFullTry() and UnsetPoolFullTry()
SetPoolFullTry() implements rados_set_pool_full_try() and
UnsetPoolFullTry() implements rados_unset_pool_full_try() octopus onwards.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Mudit Agarwal
4955746fb2 rados: add SetPoolFullTry() and UnsetPoolFullTry() for nautilus
SetPoolFullTry implements rados_set_osdmap_full_try() and
UnsetPoolFullTry implements rados_unset_osdmap_full_try() in
nautilus.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Mudit Agarwal
39278e1dd0 rados: changed build tag in rados/ioctx_nautilus.go
changed build tag in rados/ioctx_nautilus.go and
rados/ioctx_nautilus_test.go in a way such that the
functions in these files are built for nautilus only.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Mudit Agarwal
76e4089806 rados: move GetNamespace() to rados/ioctx.go
Now, we don't support ceph versions < nautilus this function
can move to a generic file so that nautilus specific functions
can be written here. Moved GetNamespace() and the corresponding
test function.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Niels de Vos
92b808cbbe rbd: add DeepCopy()
With rbd_deep_copy() an image can be copied with selected options. This
can be used to create a copy and flatten the image in one go.

Updates: ceph/ceph-csi#2077
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-01 14:40:37 +00:00
John Mulligan
e0e56486b0 makefile: add new rbd/admin package to build and ci script
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
e78058504d rbd admin: add a test to cover the Status function
This is a complex test as it requires the pool and an image before it
will return any (non empty) data.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
5a05449609 rbd admin: add tests for the simpler snap sched related functions
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
cd69fc6031 rbd admin: add mirror snapshot scheduling functions
These are the functions that let one administrate the snapshot schedules
for mirrored pools and images.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
ac2026b118 rbd admin: add tests for the LevelSpec type
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
b242e6b449 rbd admin: add initial basic types
Adds types to work with the ceph cluster (the admin type) and a type
to work with the "level spec" - which is unfortunately not well
described in the ceph docs, but I think I worked it out well enough
by reading the ceph sources.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
b6f36475b2 rbd admin: establish a new rbd/admin subpackage
This package will be used for issuing json commands to the mgr (or
mons, etc) that perform background tasks to administrate rbd pools,
images, etc.

This commit just adds a doc.go as a stub.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
274fc16d95 internal commands: swap over to common subpkg for interface types
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
bbd88f0b85 cephfs admin: swap from internal package to common packge for interfaces
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
fa8575ac26 doc: note the new "common/" subdir to the doc.go file
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan
e8e1d722c8 common commands: add new sub-package for common interface types
Add new common interface types for working with Ceph's "JSON commands"
to a public sub-package.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
oct28-yjkim
8e4b8349aa rbd: add SetSnapshot (re)implementing rbd_snap_set
Signed-off-by: oct28.yjkim <oct28.yjkim@gmail.com>
2021-06-01 10:28:25 +00:00
John Mulligan
97eefba287 rbd: add SetMirrorSiteName and GetMirrorSiteName functions
* Add SetMirrorSiteName implementing rbd_mirror_site_name_set
* Add GetMirrorSiteName implementing rbd_mirror_site_name_get

These functions are used to set the name of the site (for an entire
cluster) for rbd mirroring purposes. It's a bit strange that it's
part of rdb and not rados, but such is life.

Tests included.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 09:56:02 +00:00
Sven Anderson
1c719b199e tests: add benchmarks for PtrGuard related types
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-06-01 09:19:59 +00:00
Sébastien Han
6be8d370cb rgwadmin: add support for RadosGW Admin Ops API
Following this discussion #492

This commit introduces a new package "rgw/admin" which helps you interact
with the [RadosGW Admin Ops API](https://docs.ceph.com/en/latest/radosgw/adminops).
Not all the API capabilities are covered by this commit, but this is a
solid foundation for adding code on top. I'm expecting a few more
iterations to make 100% complete. Also, the RadosGW Admin API is going
to implement new functions soon (like bucket creation). So this library
will live on and keep catching up.

As many unit tests as possible have been added. A new integration test
suite also runs. The "micro-osd.sh" now deploys a RGW and the
integration suite tests on it. Thus the CI should cover it.

Shout out to @QuentinPerez and @IrekFasikhov for their existing
libraries. They were a very good inspiration to get started.

Co-authored-by: Irek Fasikhov <malmyzh@gmail.com>
Co-authored-by: Quentin Perez <qperez42@gmail.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 17:45:31 +02:00
Sébastien Han
edd90d8de4 make: set hostname for "test-container" test
We now set the hostname for the "test-container" to "test_ceph_aio"
where "aio" means "all in one". This will be useful for the rgw test
where we need an hostname to reach out to the endpoint.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 17:45:31 +02:00
Sven Anderson
78ff00f50b tests: update packages to keep ceph common and libraries in sync
Sometimes the version in the RPM repository don't match the version
of the base image. This change updates the packages before installing
the libraries in order to make them match.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-31 17:00:08 +02:00
Sven Anderson
c74cbb1dfe rbd: fix argument type of rbd_writesame call
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-19 13:38:06 +00:00
Niels de Vos
78f2f30f63 github: require Pacific and dpulls to pass before merging
Tests are running against Ceph Pacific and passing, so this can be
marked as a required CI status.

dpulls handles dependencies between PRs, once the status is marked with
success, the requirements have all been merged. Prevent merging when
dpulls is in a failed state.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-19 02:15:55 +02:00
John Mulligan
8ed302c7a0 entrypoint.sh: add a vim modeline to help set formatting parameters
For better or worse, unlike Go, shell does not have a standard formatting style.
Add a vim modeline that should match the basic current formatting style of the
scripts. This ought to help fellow vim users when editing the scripts.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-05-17 14:24:14 +00:00
RAJAT SINGH
512d1edd8f add build.octopus to gitignore
a file name .build.octopus gets created everytime the repoository is opened into the dev cointainer, adding it in gitignore will solve this issue.

Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
2021-05-07 10:50:35 -04:00
RAJAT SINGH
290fbb6ed1 cephfs: add wrapper for ceph_lchown
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
2021-05-05 13:44:08 +00:00
John Mulligan
d37951f7cd rbd: add MirrorImageStatusSummary implementing rbd_mirror_image_status_summary
The MirrorImageStatusSummary returns a map of image mirroring states to
the number of images in those states or an error.
Tests of both the basic conditions and actual mirrored images are added.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-05-03 16:55:10 +00:00
Sven Anderson
c45fa95b32 rados: remove pointer arithmetic on C-buffers
This change uses slices on top of C allocated array memory in order
to have a simple (no pointer arithmetic) and safe (boundary-checked)
access to its elements.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 16:20:34 +00:00
John Mulligan
986dcab058 makefile: use "octopus" as our default ceph version
We're adding a fair amount of octopus (and later) only features now, and
at the time of this writing, pacific is already out. Stop defaulting to
nautilus and default to octopus instead.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-05-03 15:37:43 +00:00
Sven Anderson
0381a092d2 tests: fix tests of the response package
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 15:07:32 +00:00
Sven Anderson
93a4fe6d61 tests: make sure that all tests are run
Some tests were not run, because entrypoint.sh used a fixed list of package,
which is easily forgotten to update.  This change creates the list of
packages dynamically.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 15:07:32 +00:00
Sven Anderson
fff2dba3fd contrib: add configuration for devcontainer
This change adds configurations for running the development
environment in a container, as supported by VSCode or GitHub
Codespaces.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 10:38:17 -04:00
John Mulligan
5964821ec0 rbd: update build tags from "octopus" to "not nautilus"
Unless these apis are deprecated in the (far?) future it is safer to say
"not nautilus" rather than explicitly tag all current and future ceph
release codenames.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-30 15:23:40 +02:00
John Mulligan
1f02b52df2 cephfs admin: update test helper functions to work on ceph "pacific"
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-30 15:23:40 +02:00
John Mulligan
52ef909e2b workflows: enable running ci against ceph pacific
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-30 15:23:40 +02:00
John Mulligan
e76404c6aa readme: document v0.9.0 and the ceph versions it supports
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:34:44 +00:00
John Mulligan
e5accf5dfc readme: no more need to document cli args needed to run fuse
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan
90e5af86ef containers: drop the unused rbd-fuse from the install packages
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan
7c19066bed makefile: remove the use of fuse from multi container tests
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan
cf29cdbb27 entrypoint.sh: replace uses of rbd-fuse with rbd command line tool
Replace uses of the rbd-fuse virtual file system with command line
invocations of `rbd import` and `rbd export`. This allows us to drop
the dependency on rbd-fuse (and fuse entirely, again). Fuse requires
additional privs and special stuff so not using it makes running the
container simpler and more secure.

The only downside that I see to this change is that `rbd import` refuses
to overwrite existing images. This means the logic of the setup
function needed to be slightly changed but I don't consider it all
that different.

I did also slightly decrease the size of the image to ~500K. I didn't
think we needed to be copying so much zeros over the wire nor did
mirroring a lot of zeros demonstrate much IMO.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan
312e4cf6b5 Makefile: allow passing arguments to the test container
Allow passing additional arguments for the entrypoint.sh script to
the test container. This is useful to limit the test run to
a single package with `--test-pkg=<X>` for example.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-19 14:01:58 -04:00
John Mulligan
fc9bd71eb3 Makefile: workaround for running dual containers with podman
When running the new test-multi-container make rule, I found that
podman generated "random" hex based hostnames for the containers.
These names work internally and do resolve but I found that putting
these in the ceph config files does not work as ceph sees them as
addresses, not hostnames, presumably due to the hex.
This change simply specifies the hostnames manually on the cli
so that both podman and docker should match docker's default behavior.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-19 14:01:58 -04:00