Commit Graph

1033 Commits

Author SHA1 Message Date
Sébastien Han 42a9620c93 rgw/admin: add ability to mock HTTP Client
We can now use a mocked client to be consumed by unit tests as well as
various users of the API.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-21 05:11:49 +00:00
Sébastien Han 53b3842e29 rgw/admin: add an interface around the HTTP Client
We now have a new interface `HTTPClient` which helps us doing various
operations such as mutating and mocking the HTTP Client.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-21 05:11:49 +00:00
Sven Anderson 5da5db0b9c devcontainer: add option to start ceph nodes for tests 2021-07-19 17:07:42 -04:00
Sébastien Han 7881ee6829 rgw/admin: use http verbs from the http package
The http package provides constants for common http methods so let's use
them.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-07-13 14:42:54 +00:00
John Mulligan 7f34da0592 cephfs: add test case for ReadConfigFile
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-07-13 13:53:48 +00:00
John Mulligan 5b727cf7a0 cephfs: add ReadConfigFile implementing ceph_conf_read_file
Unlike ReadDefaultConfigFile, ReadConfigFile supports user specified
paths.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-07-13 13:53:48 +00:00
John Mulligan 3add450741 cephfs: fix function doc comment for ReadDefaultConfigFile
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-07-13 13:53:48 +00:00
John Mulligan c009f6d19e makefile: add a flag to run go commands via a container
By running `make USE_GOCO=1` the makefile will run go build, etc.
commands using _GO_ from a _COntainer_. This is inspired by recent work
to enable development containers for VSCode, but is more aimed at
vim/cli users like myself who want to compile the code via makefile
rules and make use of vim's quickfix feature without having to have ceph
libs or, more importantly, the right version of the ceph libs, locally.

Full example: `make build test-bins USE_CACHE=1 USE_GOCO=1`

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-07-08 10:36:19 -04:00
John Mulligan 7abb5fd321 makefile: use variables for go and gofmt commands
There are use cases where the default go commands are not what you want,
including an upcoming change. This simply "variablizes" the makefile.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-07-08 10:36:19 -04:00
John Mulligan e0727d9cf7 rados: naming conventions: fix c_entry, c_namespace
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_entry -> cEntry" rados/object_iter.go
Command: gofmt -w -r "c_namespace -> cNamespace" rados/object_iter.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan f25ee98e0a rados: naming conventions: fix c_cluster_name, c_name
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_cluster_name -> cClusterName" rados/rados.go
Command: gofmt -w -r "c_name -> cName" rados/rados.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan 7f77d82bab rados: naming conventions: c_user -> cUser
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_user -> cUser" rados/rados.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan 801a9cc498 rados: naming conventions: fix c_major, c_minor, c_patch
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_major -> cMajor" rados/rados.go
Command: gofmt -w -r "c_minor -> cMinor" rados/rados.go
Command: gofmt -w -r "c_patch -> cPatch" rados/rados.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan 4c12ff0fda rados: naming conventions: c_stat -> cStat
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_stat -> cStat" rados/conn.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan 9619de1d84 rados: naming conventions: c_opt/c_val -> cOpt/cVal
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_opt -> cOpt" rados/conn.go
Command: gofmt -w -r "c_val -> cVal" rados/conn.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan debaac7031 rados: naming conventions: c_id -> cid
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_id -> cid" rados/conn.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan 46c860c614 rados: naming conventions: c_pool -> cPool
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_pool -> cPool" rados/conn.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan 5cbf7c6514 rados: naming conventions: c_path -> cPath
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_path -> cPath" rados/conn.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
John Mulligan d7f73e112d rados: naming conventions: c_name -> cName
Fix up variable names that don't meet Go standards.

Command: gofmt -w -r "c_name -> cName" rados/conn.go

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-28 13:51:01 -04:00
Sven Anderson cc08cab257 devcontainer: simplify and remove Dockerfile
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-06-25 15:03:52 +02:00
John Mulligan 4298c2bd0e docs: add an initial development guide
OK, I may have been too wordy, but its a start.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-21 09:11:36 -04:00
John Mulligan ba2e8674de readme: document v0.10.0 and the ceph versions it supports
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-14 11:21:14 -04:00
Sven Anderson 5cb1f79d74 tests: add USE_CACHE variable to enable go packages cache
If the make variable USE_CACHE is set, the test container will mount
the /go directory from a named volume that is reused in following
test runs, so that go dependencies don't have to be installed again.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-06-14 14:16:42 +00:00
RAJAT SINGH 5c49aab11e rbd: add GetMirrorUUID function
* Add GetMirrorUUID implementing rbd_mirror_uuid_get

This function is be used to get the mirroring uuid for the pool.
Basic tests included.

Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
2021-06-08 14:01:09 +00:00
John Mulligan 24eeb9512c rbd: add a MirrorImageGlobalStatusIter implementing rbd_mirror_image_global_status_list
This adds the function call in the style of an iterator, as the number
of mirrored images in the entire pool could be large. Tests are
included.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-08 12:20:53 +00:00
Sébastien Han 8179bd4437 rgw/admin: stop returning pointers
Since we are not passing a pointer, let's not return a pointer either.
Also, our use case fits well the non-returning pointer approach since
the content of the struct type is not expected to be modified.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
Sébastien Han a2c50e28a5 rgw/admin: add quota support
This commit introduces support for user and bucket quota on the rgw
admin ops API.

Co-authored-by: Irek Fasikhov malmyzh@gmail.com
Co-authored-by: Quentin Perez qperez42@gmail.com
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
Sébastien Han 7b585cc9b4 rgw/admin: add new error for SignatureDoesNotMatch
It indicates a mal-formed query for the API where some parameters are
incorrect. This is unlikely to happen with good integration test but we
never know.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
Sébastien Han cf09833810 rgw/admin: remove unnecessary url.Parse() call
http.NewRequestWithContext() already calls url.Parse() internally so we
don't need call it.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-06-07 12:23:46 -04:00
John Mulligan 2be2128e33 rbd: add CreateMirrorPeerBootstrapToken & ImportMirrorPeerBootstrapToken functions
* Add CreateMirrorPeerBootstrapToken implementing rbd_mirror_peer_bootstrap_create
* Add ImportMirrorPeerBootstrapToken implementing rbd_mirror_peer_bootstrap_import

These functions can be used to set up mirroring between pools. Basic tests
included. The tests only verify that functions work, not that they
actually mirror data. That is a job for another day.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-03 14:42:42 +00:00
Mudit Agarwal 444050570c rados: add SetPoolFullTry() and UnsetPoolFullTry()
SetPoolFullTry() implements rados_set_pool_full_try() and
UnsetPoolFullTry() implements rados_unset_pool_full_try() octopus onwards.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Mudit Agarwal 4955746fb2 rados: add SetPoolFullTry() and UnsetPoolFullTry() for nautilus
SetPoolFullTry implements rados_set_osdmap_full_try() and
UnsetPoolFullTry implements rados_unset_osdmap_full_try() in
nautilus.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Mudit Agarwal 39278e1dd0 rados: changed build tag in rados/ioctx_nautilus.go
changed build tag in rados/ioctx_nautilus.go and
rados/ioctx_nautilus_test.go in a way such that the
functions in these files are built for nautilus only.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Mudit Agarwal 76e4089806 rados: move GetNamespace() to rados/ioctx.go
Now, we don't support ceph versions < nautilus this function
can move to a generic file so that nautilus specific functions
can be written here. Moved GetNamespace() and the corresponding
test function.

Signed-off-by: Mudit Agarwal <muagarwa@redhat.com>
2021-06-02 06:16:01 +00:00
Niels de Vos 92b808cbbe rbd: add DeepCopy()
With rbd_deep_copy() an image can be copied with selected options. This
can be used to create a copy and flatten the image in one go.

Updates: ceph/ceph-csi#2077
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-01 14:40:37 +00:00
John Mulligan e0e56486b0 makefile: add new rbd/admin package to build and ci script
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan e78058504d rbd admin: add a test to cover the Status function
This is a complex test as it requires the pool and an image before it
will return any (non empty) data.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan 5a05449609 rbd admin: add tests for the simpler snap sched related functions
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan cd69fc6031 rbd admin: add mirror snapshot scheduling functions
These are the functions that let one administrate the snapshot schedules
for mirrored pools and images.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan ac2026b118 rbd admin: add tests for the LevelSpec type
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan b242e6b449 rbd admin: add initial basic types
Adds types to work with the ceph cluster (the admin type) and a type
to work with the "level spec" - which is unfortunately not well
described in the ceph docs, but I think I worked it out well enough
by reading the ceph sources.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan b6f36475b2 rbd admin: establish a new rbd/admin subpackage
This package will be used for issuing json commands to the mgr (or
mons, etc) that perform background tasks to administrate rbd pools,
images, etc.

This commit just adds a doc.go as a stub.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan 274fc16d95 internal commands: swap over to common subpkg for interface types
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan bbd88f0b85 cephfs admin: swap from internal package to common packge for interfaces
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan fa8575ac26 doc: note the new "common/" subdir to the doc.go file
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan e8e1d722c8 common commands: add new sub-package for common interface types
Add new common interface types for working with Ceph's "JSON commands"
to a public sub-package.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
oct28-yjkim 8e4b8349aa rbd: add SetSnapshot (re)implementing rbd_snap_set
Signed-off-by: oct28.yjkim <oct28.yjkim@gmail.com>
2021-06-01 10:28:25 +00:00
John Mulligan 97eefba287 rbd: add SetMirrorSiteName and GetMirrorSiteName functions
* Add SetMirrorSiteName implementing rbd_mirror_site_name_set
* Add GetMirrorSiteName implementing rbd_mirror_site_name_get

These functions are used to set the name of the site (for an entire
cluster) for rbd mirroring purposes. It's a bit strange that it's
part of rdb and not rados, but such is life.

Tests included.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 09:56:02 +00:00
Sven Anderson 1c719b199e tests: add benchmarks for PtrGuard related types
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-06-01 09:19:59 +00:00
Sébastien Han 6be8d370cb rgwadmin: add support for RadosGW Admin Ops API
Following this discussion #492

This commit introduces a new package "rgw/admin" which helps you interact
with the [RadosGW Admin Ops API](https://docs.ceph.com/en/latest/radosgw/adminops).
Not all the API capabilities are covered by this commit, but this is a
solid foundation for adding code on top. I'm expecting a few more
iterations to make 100% complete. Also, the RadosGW Admin API is going
to implement new functions soon (like bucket creation). So this library
will live on and keep catching up.

As many unit tests as possible have been added. A new integration test
suite also runs. The "micro-osd.sh" now deploys a RGW and the
integration suite tests on it. Thus the CI should cover it.

Shout out to @QuentinPerez and @IrekFasikhov for their existing
libraries. They were a very good inspiration to get started.

Co-authored-by: Irek Fasikhov <malmyzh@gmail.com>
Co-authored-by: Quentin Perez <qperez42@gmail.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 17:45:31 +02:00