diff --git a/docs/how-to/ccv0.sh b/docs/how-to/ccv0.sh index 1b64eee95..8d36639c4 100755 --- a/docs/how-to/ccv0.sh +++ b/docs/how-to/ccv0.sh @@ -88,6 +88,7 @@ Commands: - open_kata_console: Stream the kata runtime's console - open_kata_shell: Open a shell into the kata runtime - agent_pull_image: Run PullImage command against the agent with agent-ctl +- shim_pull_image: Run PullImage command against the shim with ctr - agent_create_container: Run CreateContainer command against the agent with agent-ctl - test: Test using kata with containerd - test_capture_logs: Test using kata with containerd and capture the logs in the user's home directory @@ -399,11 +400,17 @@ agent_pull_image() { run_agent_ctl_command "PullImage image=${PULL_IMAGE} cid=${CONTAINER_ID} source_creds=${SOURCE_CREDS}" } - agent_create_container() { run_agent_ctl_command "CreateContainer cid=${CONTAINER_ID}" } +shim_pull_image() { + get_ids + ctr_shim_command="ctr --namespace k8s.io shim --id ${sandbox_id} pull-image ${PULL_IMAGE} ${CONTAINER_ID}" + echo "Issuing command '${ctr_shim_command}'" + ${ctr_shim_command} +} + main() { while getopts "dh" opt; do case "$opt" in @@ -491,10 +498,13 @@ main() { agent_create_container) agent_create_container ;; + shim_pull_image) + shim_pull_image + ;; *) usage 1 ;; esac } -main $@ \ No newline at end of file +main $@ diff --git a/docs/how-to/how-to-build-and-test-ccv0.md b/docs/how-to/how-to-build-and-test-ccv0.md index 82a7012d2..cbe8591c8 100644 --- a/docs/how-to/how-to-build-and-test-ccv0.md +++ b/docs/how-to/how-to-build-and-test-ccv0.md @@ -2,34 +2,61 @@ ## Introduction and Background -In order to try and make building (locally) and demoing the Kata Containers `CCv0` code base as simple as possible I've shared a script [`ccv0.sh`](./ccv0.sh). This script was originally my attempt to automate the steps of the [Developer Guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md) so that I could do different sections of them repeatedly and reliably as I was playing around with make changes to different parts of the Kata code base. I then tried to weave in some of the [`tests/.ci`](https://github.com/kata-containers/tests/tree/main/.ci) scripts in order to have less duplicated code and to make it support for platforms. Finally I extended it to include some calls to start kata pods in Kubernetes and call [`agent-ctl`](https://github.com/kata-containers/kata-containers/tree/main/tools/agent-ctl) to test the agent endpoint for pull image on guest for the CCv0 roadmap. +In order to try and make building (locally) and demoing the Kata Containers `CCv0` code base as simple as possible I've +shared a script [`ccv0.sh`](./ccv0.sh). This script was originally my attempt to automate the steps of the +[Developer Guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md) so that I could do +different sections of them repeatedly and reliably as I was playing around with make changes to different parts of the +Kata code base. I then tried to weave in some of the [`tests/.ci`](https://github.com/kata-containers/tests/tree/main/.ci) +scripts in order to have less duplicated code and to make it support for platforms. Finally I extended it to include some +calls to start kata pods in Kubernetes and call +[`agent-ctl`](https://github.com/kata-containers/kata-containers/tree/main/tools/agent-ctl) and [`ctr shim`] to test the +shim and agent endpoint for pull image on guest for the CCv0 roadmap. -At the time of writing we only have some basic Kata agent pull image support for CCv0 included into the [`CCv0` branch](https://github.com/kata-containers/kata-containers/tree/CCv0), so the testing is limited to this, but as more functionality is added I'm hoping that this script can grow and expand to handle it. +At the time of writing we only have some basic Kata agent and containerd shim support for image offload included into the +[`CCv0` branch](https://github.com/kata-containers/kata-containers/tree/CCv0), so the testing is limited to this, but as +more functionality is added I'm hoping that this script can grow and expand to handle it. -*Disclaimer: This script has mostly just been used and tested by me ([@stevenhorsman](https://github.com/stevenhorsman)), so there might be issues with it. I'm happy to try and help solve these if possible, but this shouldn't be considered a fully supported process by the Kata Containers community.* +*Disclaimer: This script has mostly just been used and tested by me ([@stevenhorsman](https://github.com/stevenhorsman)),* +*so there might be issues with it. I'm happy to try and help solve these if possible, but this shouldn't be considered a* +*fully supported process by the Kata Containers community.* ## Basic demo How-to -In order to build, and demo the CCv0 functionality, these are the steps I take: +In order to build, and demo the CCv0 functionality, these are the steps I take (*Note: I've only tested this script* +*running as root*): - Provision a new VM - - *I choose a Ubuntu 20.04 8GB VM for this as I had one available. I think that the only part of the script that is OS dependent is the install of `git`, `socat` and `qemu-utils`(optional) using apt-get to bootstrap the rest of the installs. In order to run this on any platform just use your package manager to install these before running the `ccv0.sh` script and comment out the apt-get line with `sudo sed -i -e 's/\(sudo apt-get update .*\)$/# \1/g' ccv0.sh`*. -- Copy the script over to your VM *(I put it in the home directory)* and ensure it has execute permission by running `chmod u+x ccv0.sh` + - *I choose a Ubuntu 20.04 8GB VM for this as I had one available. There are some dependences on apt-get installed* + *packages, so these will need re-working to be compatible with other platforms.* +- Copy the script over to your VM *(I put it in the home directory)* and ensure it has execute permission by running +`chmod u+x ccv0.sh` - Optionally set up some environment variables - - By default the script checks out the `CCv0` branches of the `kata-containers/kata-containers` and `kata-containers/tests` repositories, but it is designed to be used to test of personal forks and branches as well. If you want to build and run these you can export the `katacontainers_repo`, `katacontainers_branch`, `tests_repo` and `tests_branch` variables e.g. `export katacontainers_repo=github.com/stevenhorsman/kata-containers && export katacontainers_branch=stevenh/agent-pull-image-endpoint && export tests_repo=github.com/stevenhorsman/tests && export tests_branch=stevenh/add-ccvo-changes-to-build` before running the script. + - By default the script checks out the `CCv0` branches of the `kata-containers/kata-containers` and + `kata-containers/tests` repositories, but it is designed to be used to test of personal forks and branches as well. + If you want to build and run these you can export the `katacontainers_repo`, `katacontainers_branch`, `tests_repo` + and `tests_branch` variables e.g. `export katacontainers_repo=github.com/stevenhorsman/kata-containers && export katacontainers_branch=stevenh/agent-pull-image-endpoint && export tests_repo=github.com/stevenhorsman/tests && export tests_branch=stevenh/add-ccvo-changes-to-build` + before running the script. - Run the full build process with `. ~/ccv0.sh -d build_and_install_all` - - *I run this script sourced just so that the required installed components are accessible on the `PATH` to the rest of the process without having to reload the session.* + - *I run this script sourced just so that the required installed components are accessible on the `PATH` to the rest* + *of the process without having to reload the session.* - The steps that `build_and_install_all` takes is: - - Checkout the git repos for the `tests` and `kata-containers` repos as specified by the environment variables (default to `CCv0` branches if they are not supplied) + - Checkout the git repos for the `tests` and `kata-containers` repos as specified by the environment variables + (default to `CCv0` branches if they are not supplied) - Use the `tests/.ci` scripts to install the build dependencies - Build and install the Kata runtime - - Configure Kata to use containerd and for debug to be enabled (including enabling console access to the kata-runtime, which should only be done in development) - - Create, build and install a rootfs for the Kata hypervisor to use. For 'CCv0' this is currently based on Fedora 34 and has extra packages like `skopeo` and `umoci` added. + - Configure Kata to use containerd and for debug to be enabled (including enabling console access to the + kata-runtime, which should only be done in development) + - Create, build and install a rootfs for the Kata hypervisor to use. For 'CCv0' this is currently based on Ubuntu + 20.04 and has extra packages like `skopeo` and `umoci` added. - Build the Kata guest kernel - Install QEMU - Set up `agent-ctl` testing by building the binary and configuring a bundle directory for it - Initialising Kubernetes to use the VM as a single node cluster - - The first time this runs it may take a while, but subsequent runs will be quicker as more things are already installed and they can be further cut down by not running all the above steps [see "Additional script usage" below](#additional-script-usage) - - *Depending on how where your VMs are and how IPs are shared you might possibly get an error during "Store custom stress image in registry" from docker matching `ERROR: toomanyrequests: Too Many Requests`. In order to get around this log into docker hub with `sudo docker login` and re-run the step with `. ~/ccv0.sh -d init_kubernetes`.* + - The first time this runs it may take a while, but subsequent runs will be quicker as more things are already + installed and they can be further cut down by not running all the above steps + [see "Additional script usage" below](#additional-script-usage) + - *Depending on how where your VMs are and how IPs are shared you might possibly get an error during "Store custom* + *stress image in registry" from docker matching `ERROR: toomanyrequests: Too Many Requests`. In order to get around* + *this log into docker hub with `docker login` and re-run the step with `. ~/ccv0.sh -d init_kubernetes`.* - Check that your Kubernetes cluster has been correctly set-up: ``` $ kubectl get nodes @@ -49,7 +76,8 @@ $ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-kata 1/1 Running 0 29s ``` -- Create a new terminal to the VM and open shell into kata container and check the `/run/kata-containers` directory doesn't have a bundle unpack for container id `0123456789`: +- Create a new terminal to the VM and open shell into kata container and check the `/run/kata-containers` directory +doesn't have a bundle unpack for container id `0123456789`: ``` $ ~/ccv0.sh -d open_kata_shell bash-5.1# ls -al /run/kata-containers/ @@ -65,13 +93,44 @@ drwxr-xr-x 3 root root 60 Sep 6 09:44 shared ``` $ ~/ccv0.sh -d open_kata_console ``` -- In the first terminal run the pull image agent endpoint: +- In the first terminal run the pull image on guest command against the Kata agent, via the shim (`containerd-shim-kata-v2`). +This can be achieved using the [containerd](https://github.com/containerd/containerd) CLI tool, `ctr`, which can be used to +interact with the shim directly. The command takes the form +`ctr --namespace k8s.io shim --id pull-image ` and can been run directly, or through +the `ccv0.sh` script to automatically fill in the variables: - Optionally set up some environment variables to set the image and credentials used: - - By default the agent pull test in `ccv0.sh` will use the image `registry.fedoraproject.org/fedora:latest` which requires no authentication. If you want to use a different image, first set the `PULL_IMAGE` environment variable e.g. `export PULL_IMAGE="docker.io/library/busybox:latest"`. - - If the container registry for the image requires authentication then this can be set with an environment variable `SOURCE_CREDS`. For example to use `docker.io` as an authenticated user first run `export SOURCE_CREDS=":"` - - *Note: the credentials support on the agent request is a tactical solution for the short-term proof of concept to allow more images to be pulled and tested. Once we have support for getting keys into the kata guest using the attestation-agent and/or KBS I'd expect container registry credentials to be looked up using that mechanism.* - - Run the pull image agent endpoint with `~/ccv0.sh -d agent_pull_image`: - - *For unknown reasons sometimes the unpack fails the first time and the sandbox crashes, but seems to work the second time and the pod will restart automatically, so just re-open the shell and console and re-run the agent_pull_image.* + - By default the shim pull image test in `ccv0.sh` will use the image `registry.fedoraproject.org/fedora:latest` + which requires no authentication. If you want to use a different image, first set the `PULL_IMAGE` environment + variable e.g. `export PULL_IMAGE="docker.io/library/busybox:latest"`. Currently the containerd shim pull image + code doesn't support using a container registry that requires authentication, so if this is required, see the + below steps to run the pull image command against the agent directly. + - Run the pull image agent endpoint with `~/ccv0.sh -d shim_pull_image`: + - *For reasons, we think are related to the disk space remaining when unpacking the image bundle, sometimes the* + *[unpack fails the first time and the sandbox crashes](https://github.com/kata-containers/kata-containers/issues/2876),* + *but seems to work the second time and the pod will restart automatically, so just re-open the shell and console and* + *re-run the shim_pull_image.* +```bash +$ ~/ccv0.sh shim_pull_image +Issuing command 'ctr --namespace k8s.io shim --id 4cabb081a0d90e0d041e433eebf315983ef2a945f1b1759dea51ea41ceff3ac8 pull-image registry.fedoraproject.org/fedora:latest 0123456789' +``` + - Alternatively you can issue the command directly to the kata-agent pull image endpoint, which also supports + credentials in order to pull from an authenticated registry: + - Optionally set up some environment variables to set the image and credentials used: + - Set the `PULL_IMAGE` environment variable e.g. `export PULL_IMAGE="docker.io/library/busybox:latest"` + if a specific container image is required. + - If the container registry for the image requires authentication then this can be set with an environment + variable `SOURCE_CREDS`. For example to use `docker.io` as an authenticated user first run + `export SOURCE_CREDS=":"` + - *Note: the credentials support on the agent request is a tactical solution for the short-term* + *proof of concept to allow more images to be pulled and tested. Once we have support for getting* + *keys into the kata guest using the attestation-agent and/or KBS I'd expect container registry* + *credentials to be looked up using that mechanism.* + - Run the pull image agent endpoint with `~/ccv0.sh -d agent_pull_image`: + - *For reasons, we think are related to the disk space remaining when unpacking the image bundle,* + *sometimes the* + *[unpack fails the first time and the sandbox crashes](https://github.com/kata-containers/kata-containers/issues/2876),* + *but seems to work the second time and the pod will restart automatically, so just re-open the shell and console* + *and re-run the agent_pull_image.* ``` $ ~/ccv0.sh -d agent_pull_image Finished release [optimized] target(s) in 0.21s @@ -188,7 +247,14 @@ pod "nginx-kata" deleted ## Additional script usage -As well as being able to use the script as above to build all of kata-containers from scratch it can be used to just re-build bits of it by running the script with different parameters. For example after the first build you will often not need to re-install the dependencies, QEMU or the Guest kernel, but just test code changes made to the runtime and agent. This can be done by running `. ~/ccv0.sh -d rebuild_and_install_kata` (*Note this re-does the checkout from git, do you changes are only made locally it is better to do the individual steps e.g. `. ~/ccv0.sh -d build_kata_runtime && . ~/ccv0.sh -d build_and_add_agent_to_rootfs && . ~/ccv0.sh -d build_and_install_rootfs`).* There are commands for a lot of steps in building, setting up and testing and the full list can be seen by running `~/ccv0.sh help`: +As well as being able to use the script as above to build all of kata-containers from scratch it can be used to just +re-build bits of it by running the script with different parameters. For example after the first build you will often +not need to re-install the dependencies, QEMU or the Guest kernel, but just test code changes made to the runtime and +agent. This can be done by running `. ~/ccv0.sh -d rebuild_and_install_kata`. (*Note this does a hard checkout* +*from git, so if your changes are only made locally it is better to do the individual steps e.g.* +`. ~/ccv0.sh -d build_kata_runtime && . ~/ccv0.sh -d build_and_add_agent_to_rootfs && . ~/ccv0.sh -d build_and_install_rootfs`). +There are commands for a lot of steps in building, setting up and testing and the full list can be seen by running +`~/ccv0.sh help`: ``` $ ~/ccv0.sh help Overview: