ceph/ContainerBuild.md

Build Ceph within a Container

The Ceph project includes a script and additional files that help build the ceph.git sources inside an OCI-style container. This script requires Python 3 (tested with 3.8 or later) and Podman or Docker.

The script aims to make it simple for a developer or anyone wanting to compile Ceph on Linux but not wanting to dedicate a full physical or virtual host to the job. The containers encapsulate the build dependencies and environment and are quick and easy to clean up when you no longer need them.

build-with-containers.py Introduction

The script exists within the ceph.git tree at src/script/build-with-container.py. The script operates on the entire Ceph repository and will automatically try to detect the root of the ceph.git tree. At any time the --help option can be passed to the script for a complete listing of command line options.

To start with the default options run ./src/script/build-with-container.py. The script will first check if a build image already exists, if not it will construct one using the Dockerfile.build file in the Ceph tree [1].

Once a build container is available the script will build Ceph from source using a build directory named build in the root of the source tree. The default environment uses a base distribution of centos9.

You can select the base distribution to use with the --distro/-d option. The list of available distribution bases include ubuntu22.04 and centos9. These choices are known to work well. Other more experimental choices, including ubuntu24.04 and centos10, are available as well but do not expect these platforms to work without tinkering. This option is a shorthand for specifying the base container image that will be used to construct the build image, the tags used for that image, as well as helping determine what kind of packages can be produced using that build image.

You can specify where build artifacts are written with the --build-dir/-b option. For example ./src/script/build-with-container.py -b build.try1.

The tool supports mutliple build targets. Often these targets are chained together - for example almost all targets depend on the container target. To select a target use the --execute/-e command line option. For example: ./src/script/build-with-container.py -e tests will execute the unit tests after building them. (Note: the --no-prereqs option exists to disable the chaining but this should not be needed in most circumstances)

[1] - This behavior can be customized with the --image-sources/-I option.

Examples:

Build from source code on CentOS 9, with a build directory named for the base distribution:

./src/script/build-with-container.py -d centos9 -b build.centos9 -e build

Build from source code on Ubuntu 22.04, with a build directory named for the base distribution:

./src/script/build-with-container.py -d ubuntu22.04 -b build.u2204 -e build

Build RPM packages on Centos 9:

./src/script/build-with-container.py -d centos9 -e packages

Build Debian packages on Ubuntu 22.04 (Jammy):

./src/script/build-with-container.py -d ubuntu22.04 -e packages

Common Targets

• build - Build Ceph sources. Compiles C/C++ code, etc
• tests - Execute unit tests. Runs the unit test suite, depends on buildtests to compile some of the test suites
• custom - Execute a custom command. See description below
• packages - Build Ceph packages of the selected distribution's native package type
• interactive - Start the build container in an interactive mode

Custom Commands

The custom target can be used to run a single command that the script is not already programmed to handle. The custom command must come after a -- to terminate the normal command line arguments for build-with-container.py. For example:

./src/script/build-with-container.py -d ubuntu22.04 -e custom -- shellcheck src/script/buildcontainer-setup.sh

Interactive mode

The interactive target can be used to run a shell within the container. This is handy for those times you want to run multiple commands, by hand, within the container environment. As an example:

./src/script/build-with-container.py -d ubuntu22.04 -e interactive

Additional Features

Control build image name and tags

The build-with-container.py script automatically generates images names based on a standard name and auto-generated tag. By default the script names the image ceph-build and assumes the images are local only - the image name will not refer to any image registry. The images are tagged with the name of the current branch and base distribution. For example, assuming we're on a branch named wip-test and we execute the script with -d ubuntu22.04 we will expect or build an image named ceph-build:wip-test.ubuntu22.04.

The image name/repository can be customized using the --image-repo option. The tag can be overridden by the --tag option or extended by using the --tag option with a plus (+) character at the start of the value. For example: ./src/script/build-with-container.py --image-repo=quay.io/example/build-example --tag=foobar would create or reuse an image named quay.io/example/build-example:foobar. ./src/script/build-with-container.py --image-repo=quay.io/example/build-example --distro=centos9 --tag=+foobar would use an image named quay.io/example/build-example:wip-test.centos9.foobar.

If one wants to override the name of the branch or the branch can not be automatically detected the --current-branch option can be supplied to customize this value.

Control build image source

By default build-with-container.py will try reuse build images if they are cached in the local container store. If the image is not present it will build one. In addition to these default actions the script can be intructed to "pull" an image from a remote registry, possibly avoiding the need to build an image.

How the build image is acquired can be controlled using the --image-sources option. The option takes a comma-separated list of terms. The terms are cache, pull, and build:

• build - Create a new build image
• cache - Check for an existing image in local container storage
• pull - Pull an image form a container image registry

So for example if you did not want to fall back to building an image locally, you could pass --image-sources=cache,pull to the script. Passing --image-sources=build will force the script to rebuild an image even if would be available elsewhere.

When an image is to be built the image base is typically derived from the name of the distribution being used. However, the base image can be overridden on the command line using the --base-image option. For example, if one had a local registry with a CentOS 9 (Stream) base image the following example could be used: ./src/script/build-with-container.py -d centos9 --base-image myreg.example.com/ceph/centos-base:9

Controlling where files are written

By default the directory holding the Ceph source tree is mounted at /ceph within the container (controlled by the --homedir option). Various build tasks will write files to this directory. In some cases it's useful to keep the directory free from changes and so the --overlay-dir option can be used to make that volume use an overlay.

The overlay directory will be automatically created if needed and will contain a content directory for new or updated files and a work directory, a special directory needed by the overlayfs. For example: ./src/script/build-with-container.py --overlay-dir build.ovr -b build.inner -d centos9 will end up creating a directory build.ovr/content/build.inner which will contain the results of the compile that would typically appear in just build.inner. Other writes that would have normally effected the source tree will appear in build.ovr/content

The overlay can also be temporary, with no files persisted after the container has exited. Pass --overlay-dir=- to enable this option. Note that invoking build-with-container.py default targets may use multiple container instances and passing this option will break those targets.