Commit Graph

839 Commits

Author SHA1 Message Date
John Mulligan fa8575ac26 doc: note the new "common/" subdir to the doc.go file
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
John Mulligan e8e1d722c8 common commands: add new sub-package for common interface types
Add new common interface types for working with Ceph's "JSON commands"
to a public sub-package.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 11:06:19 +00:00
oct28-yjkim 8e4b8349aa rbd: add SetSnapshot (re)implementing rbd_snap_set
Signed-off-by: oct28.yjkim <oct28.yjkim@gmail.com>
2021-06-01 10:28:25 +00:00
John Mulligan 97eefba287 rbd: add SetMirrorSiteName and GetMirrorSiteName functions
* Add SetMirrorSiteName implementing rbd_mirror_site_name_set
* Add GetMirrorSiteName implementing rbd_mirror_site_name_get

These functions are used to set the name of the site (for an entire
cluster) for rbd mirroring purposes. It's a bit strange that it's
part of rdb and not rados, but such is life.

Tests included.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-06-01 09:56:02 +00:00
Sven Anderson 1c719b199e tests: add benchmarks for PtrGuard related types
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-06-01 09:19:59 +00:00
Sébastien Han 6be8d370cb rgwadmin: add support for RadosGW Admin Ops API
Following this discussion #492

This commit introduces a new package "rgw/admin" which helps you interact
with the [RadosGW Admin Ops API](https://docs.ceph.com/en/latest/radosgw/adminops).
Not all the API capabilities are covered by this commit, but this is a
solid foundation for adding code on top. I'm expecting a few more
iterations to make 100% complete. Also, the RadosGW Admin API is going
to implement new functions soon (like bucket creation). So this library
will live on and keep catching up.

As many unit tests as possible have been added. A new integration test
suite also runs. The "micro-osd.sh" now deploys a RGW and the
integration suite tests on it. Thus the CI should cover it.

Shout out to @QuentinPerez and @IrekFasikhov for their existing
libraries. They were a very good inspiration to get started.

Co-authored-by: Irek Fasikhov <malmyzh@gmail.com>
Co-authored-by: Quentin Perez <qperez42@gmail.com>
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 17:45:31 +02:00
Sébastien Han edd90d8de4 make: set hostname for "test-container" test
We now set the hostname for the "test-container" to "test_ceph_aio"
where "aio" means "all in one". This will be useful for the rgw test
where we need an hostname to reach out to the endpoint.

Signed-off-by: Sébastien Han <seb@redhat.com>
2021-05-31 17:45:31 +02:00
Sven Anderson 78ff00f50b tests: update packages to keep ceph common and libraries in sync
Sometimes the version in the RPM repository don't match the version
of the base image. This change updates the packages before installing
the libraries in order to make them match.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-31 17:00:08 +02:00
Sven Anderson c74cbb1dfe rbd: fix argument type of rbd_writesame call
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-19 13:38:06 +00:00
Niels de Vos 78f2f30f63 github: require Pacific and dpulls to pass before merging
Tests are running against Ceph Pacific and passing, so this can be
marked as a required CI status.

dpulls handles dependencies between PRs, once the status is marked with
success, the requirements have all been merged. Prevent merging when
dpulls is in a failed state.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-05-19 02:15:55 +02:00
John Mulligan 8ed302c7a0 entrypoint.sh: add a vim modeline to help set formatting parameters
For better or worse, unlike Go, shell does not have a standard formatting style.
Add a vim modeline that should match the basic current formatting style of the
scripts. This ought to help fellow vim users when editing the scripts.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-05-17 14:24:14 +00:00
RAJAT SINGH 512d1edd8f add build.octopus to gitignore
a file name .build.octopus gets created everytime the repoository is opened into the dev cointainer, adding it in gitignore will solve this issue.

Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
2021-05-07 10:50:35 -04:00
RAJAT SINGH 290fbb6ed1 cephfs: add wrapper for ceph_lchown
Signed-off-by: RAJAT SINGH <rajasing@redhat.com>
2021-05-05 13:44:08 +00:00
John Mulligan d37951f7cd rbd: add MirrorImageStatusSummary implementing rbd_mirror_image_status_summary
The MirrorImageStatusSummary returns a map of image mirroring states to
the number of images in those states or an error.
Tests of both the basic conditions and actual mirrored images are added.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-05-03 16:55:10 +00:00
Sven Anderson c45fa95b32 rados: remove pointer arithmetic on C-buffers
This change uses slices on top of C allocated array memory in order
to have a simple (no pointer arithmetic) and safe (boundary-checked)
access to its elements.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 16:20:34 +00:00
John Mulligan 986dcab058 makefile: use "octopus" as our default ceph version
We're adding a fair amount of octopus (and later) only features now, and
at the time of this writing, pacific is already out. Stop defaulting to
nautilus and default to octopus instead.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-05-03 15:37:43 +00:00
Sven Anderson 0381a092d2 tests: fix tests of the response package
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 15:07:32 +00:00
Sven Anderson 93a4fe6d61 tests: make sure that all tests are run
Some tests were not run, because entrypoint.sh used a fixed list of package,
which is easily forgotten to update.  This change creates the list of
packages dynamically.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 15:07:32 +00:00
Sven Anderson fff2dba3fd contrib: add configuration for devcontainer
This change adds configurations for running the development
environment in a container, as supported by VSCode or GitHub
Codespaces.

Signed-off-by: Sven Anderson <sven@redhat.com>
2021-05-03 10:38:17 -04:00
John Mulligan 5964821ec0 rbd: update build tags from "octopus" to "not nautilus"
Unless these apis are deprecated in the (far?) future it is safer to say
"not nautilus" rather than explicitly tag all current and future ceph
release codenames.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-30 15:23:40 +02:00
John Mulligan 1f02b52df2 cephfs admin: update test helper functions to work on ceph "pacific"
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-30 15:23:40 +02:00
John Mulligan 52ef909e2b workflows: enable running ci against ceph pacific
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-30 15:23:40 +02:00
John Mulligan e76404c6aa readme: document v0.9.0 and the ceph versions it supports
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:34:44 +00:00
John Mulligan e5accf5dfc readme: no more need to document cli args needed to run fuse
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan 90e5af86ef containers: drop the unused rbd-fuse from the install packages
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan 7c19066bed makefile: remove the use of fuse from multi container tests
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan cf29cdbb27 entrypoint.sh: replace uses of rbd-fuse with rbd command line tool
Replace uses of the rbd-fuse virtual file system with command line
invocations of `rbd import` and `rbd export`. This allows us to drop
the dependency on rbd-fuse (and fuse entirely, again). Fuse requires
additional privs and special stuff so not using it makes running the
container simpler and more secure.

The only downside that I see to this change is that `rbd import` refuses
to overwrite existing images. This means the logic of the setup
function needed to be slightly changed but I don't consider it all
that different.

I did also slightly decrease the size of the image to ~500K. I didn't
think we needed to be copying so much zeros over the wire nor did
mirroring a lot of zeros demonstrate much IMO.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-23 13:07:21 +00:00
John Mulligan 312e4cf6b5 Makefile: allow passing arguments to the test container
Allow passing additional arguments for the entrypoint.sh script to
the test container. This is useful to limit the test run to
a single package with `--test-pkg=<X>` for example.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-19 14:01:58 -04:00
John Mulligan fc9bd71eb3 Makefile: workaround for running dual containers with podman
When running the new test-multi-container make rule, I found that
podman generated "random" hex based hostnames for the containers.
These names work internally and do resolve but I found that putting
these in the ceph config files does not work as ceph sees them as
addresses, not hostnames, presumably due to the hex.
This change simply specifies the hostnames manually on the cli
so that both podman and docker should match docker's default behavior.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-19 14:01:58 -04:00
John Mulligan 40e1224871 rbd: add tests for GetGlobalMirrorStatus based on two ceph clusters
With the infrastructure for running two ceph clusters now available,
we add tests to ensure the functionality of GetGlobalMirrorStatus in
the more typical case, when one or more mirroring clusters exist,

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-13 09:47:43 +00:00
John Mulligan de9505f862 rbd: add a test helper to get a conn for a given cluster by config
This function can be used to connect to a non-default ceph cluster,
in a simple, convenient way similar to the function we have for
connecting to the default cluster.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-13 09:47:43 +00:00
John Mulligan f84140356a rbd: add CreateMirrorSnapshot implementing rbd_mirror_image_create_snapshot
This function is used to "manually" trigger a snapshot for snapshot
based mirroring. It is needed as a pre-req to test actual cross-cluster
mirroring and the full functionality of GetGlobalMirrorStatus.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-13 09:47:43 +00:00
John Mulligan 0946b9e0e2 entrypoint.sh: export MIRROR_CONF for use by go test code
Later changes to the test Go code will make use of the MIRROR_CONF
variable to access the secondary ceph cluster.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-13 09:47:43 +00:00
John Mulligan 5c537e20ad rbd: add GetGlobalMirrorStatus implementing rbd_mirror_image_get_global_status
This function is added, along with required return types and some helper
functions, to support fetching the global mirroring status component
that makes up a major part of `rbd mirror image status` command's
output.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-13 09:47:43 +00:00
John Mulligan dfa74510aa cephfs admin: replace mon and mgr command helpers with common versions
Remove the logic previously ported over to internal/commands and just
leave the functions in place as to not have to excessively touch the
existing code.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-07 00:25:44 +00:00
John Mulligan 55a5732b3f cephfs admin: replace "command tracer" with common version from commands
Previously, this functionality was taken into commands package. Use
that.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-07 00:25:44 +00:00
John Mulligan 20b3c151b3 cephfs admin: replace response with commands.Response
Now that we have shared support working with "command json" we attempt
a minimal conversion, using aliases where appropriate, to the
new commands.Response type.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-07 00:25:44 +00:00
John Mulligan 0b1e3f5d7e commands: port tracing helper type from cephfs/admin
This debugging code can be made common.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-07 00:25:44 +00:00
John Mulligan 6b3b0fdc46 commands: add a test file for new shared Response type
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-07 00:25:44 +00:00
John Mulligan 93676ab094 commands: port response type and related to commands pkg
Convert the response type from cephfs/admin to the new package.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-07 00:25:44 +00:00
John Mulligan 42e9904d0b commands: start a new internal sub-package for working with json commands
This code was part of cephfs/admin but will be shared with a soon-to-be
rbd/admin package.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-04-07 00:25:44 +00:00
Sven Anderson 0cc1dcb90d ci: add optional mirroring setup entrypoint.sh
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-04-06 08:59:55 -04:00
Sven Anderson b72758be55 ci: add multi-container test to workflows
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-04-06 08:59:55 -04:00
Sven Anderson 9bd143e06e ci: add test-multi-container Makefile target
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-04-06 08:59:55 -04:00
Sven Anderson 362929f8d1 ci: add wait option to entrypoint script
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-04-06 08:59:55 -04:00
Sven Anderson 3fb003d886 ci: prepare micro-osd.sh for cross-container connections
Signed-off-by: Sven Anderson <sven@redhat.com>
2021-04-06 08:59:55 -04:00
Sven Anderson 014a3f6308 ptrguard: assert pointer and uintptr have same size
Uintptr are guaranteed to be at least of the size of a pointer.  However, in
theory it could be larger.  Since the code of PtrGuard assumes both have the
same size and to be on the safe side, this change adds a compile-time check,
that the sizes of uintptr and pointers are the same.

Signed-off-by: Sven Anderson <sven@anderson.de>
2021-03-26 11:41:14 +00:00
John Mulligan e9f8b47f71 containers: use the current Go 1.15 release
The current version we are testing with is no longer supported.
This updates Go to a supported released version.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-03-24 14:00:05 +00:00
John Mulligan 78ee562863 makefile: remove /dev/fuse and SYS_ADMIN cap from test
When invoking podman/docker/etc. it is no longer needed to add fuse
support, or (AFAICT) keeping cap SYS_ADMIN.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-03-21 06:19:40 +00:00
John Mulligan 2d85e6f235 entrypoint: remove GO_CEPH_TEST_REQUIRE_MOUNT environment var
This env var is no longer used, we can drop it from the entrypoint
script.

Signed-off-by: John Mulligan <jmulligan@redhat.com>
2021-03-21 06:19:40 +00:00