kata-containers/Developer-Guide.md

• Warning
• Assumptions
• Initial setup
• Requirements to build individual components
• Build and install the Kata Containers runtime
  • Check hardware requirements
  • Configure to use initrd or rootfs image
  • Enable full debug
• Build and install Kata proxy
• Build and install Kata shim
• Create and install rootfs and initrd image
  • Build a custom Kata agent - OPTIONAL
  • Get the osbuilder
  • Create a rootfs image
    • Create a local rootfs
    • Add a custom agent to the image - OPTIONAL
    • Build a rootfs image
    • Install the rootfs image
  • Create an initrd image - OPTIONAL
    • Create a local rootfs for initrd image
    • Build an initrd image
    • Install the initrd image
• Install guest kernel images
• Run Kata Containers with Docker
  • Update Docker configuration
  • Create a container using Kata
• Troubleshoot Kata Containers
• Appendices
  • Checking Docker default runtime
  • Set up a debug console
    • Create a custom image containing a shell
    • Create a debug systemd service
    • Build the debug image
    • Configure runtime for custom debug image
    • Ensure debug options are valid
    • Create a container
    • Connect to the virtual machine using the debug console
    • Obtain details of the image

Warning

This document is written specifically for developers: it is not intended for end users.

Assumptions

• You are working on a non-critical test or development system.

Initial setup

The recommended way to create a development environment is to first install the packaged versions of the Kata Containers components to create a working system.

The installation guide instructions will install all required Kata Containers components, plus Docker*, the hypervisor, and the Kata Containers image and guest kernel.

Requirements to build individual components

You need to install the following to build Kata Containers components:

• golang version 1.8.3 or newer.

  To view the versions of go known to work, see the golang entry in the versions database.

• make.

• gcc (required for building the shim and runtime).

Build and install the Kata Containers runtime

$ go get -d -u github.com/kata-containers/runtime
$ cd $GOPATH/src/github.com/kata-containers/runtime
$ make && sudo -E PATH=$PATH make install

The build will create the following:

• runtime binary: /usr/local/bin/kata-runtime
• configuration file: /usr/share/defaults/kata-containers/configuration.toml

Check hardware requirements

You can check if your system is capable of creating a Kata Container by running the following:

$ sudo kata-runtime kata-check

If your system is not able to run Kata Containers, the previous command will error out and explain why.

Configure to use initrd or rootfs image

Kata containers can run with either an initrd image or a rootfs image.

If you want to test with initrd, make sure you have initrd = /usr/share/kata-containers/kata-containers-initrd.img in /usr/share/defaults/kata-containers/configuration.toml and comment out the image line with the following:

$ sudo sed -i 's/^\(image =.*\)/# \1/g' /usr/share/defaults/kata-containers/configuration.toml

You can create the initrd image as shown in the create an initrd image section.

If you want to test with a rootfs image, make sure you have image = /usr/share/kata-containers/kata-containers.img in /usr/share/defaults/kata-containers/configuration.toml and comment out the initrd line with the following:

$ sudo sed -i 's/^\(initrd =.*\)/# \1/g' /usr/share/defaults/kata-containers/configuration.toml

The rootfs image is created as shown in the create a rootfs image section.

One of the initrd and image options in kata runtime config file MUST be set but not both. The main difference between the options is that the size of initrd(10MB+) is significantly smaller than rootfs image(100MB+).

Enable full debug

Enable full debug as follows:

$ sudo sed -i -e 's/^# *\(enable_debug\).*=.*$/\1 = true/g' /usr/share/defaults/kata-containers/configuration.toml
$ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.log=debug"/g' /usr/share/defaults/kata-containers/configuration.toml

Build and install Kata proxy

$ go get -d -u github.com/kata-containers/proxy
$ cd $GOPATH/src/github.com/kata-containers/proxy && make && sudo make install

Build and install Kata shim

$ go get -d -u github.com/kata-containers/shim
$ cd $GOPATH/src/github.com/kata-containers/shim && make && sudo make install

Create and install rootfs and initrd image

Build a custom Kata agent - OPTIONAL

Note:

• You should only do this step if you are testing with the latest version of the agent.

$ go get -d -u github.com/kata-containers/agent
$ cd $GOPATH/src/github.com/kata-containers/agent && make

Get the osbuilder

$ go get -d -u github.com/kata-containers/osbuilder

Create a rootfs image

Create a local rootfs

As a prerequisite, you need to install Docker. Otherwise, you will not be able to run the rootfs.sh script with USE_DOCKER=true as expected in the following example.

$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs
$ sudo rm -rf ${ROOTFS_DIR}
$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true ./rootfs.sh ${distro}'

You MUST choose one of alpine, centos, clearlinux, euleros, and fedora for ${distro}.

Note:

• You must ensure that the default Docker runtime is runc to make use of the USE_DOCKER variable. If that is not the case, remove the variable from the previous command. See Checking Docker default runtime.

Add a custom agent to the image - OPTIONAL

Note:

• You should only do this step if you are testing with the latest version of the agent.

$ sudo install -o root -g root -m 0550 -t ${ROOTFS_DIR}/bin ../../agent/kata-agent
$ sudo install -o root -g root -m 0440 ../../agent/kata-agent.service ${ROOTFS_DIR}/usr/lib/systemd/system/
$ sudo install -o root -g root -m 0440 ../../agent/kata-containers.target ${ROOTFS_DIR}/usr/lib/systemd/system/

Build a rootfs image

$ cd $GOPATH/src/github.com/kata-containers/osbuilder/image-builder
$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'

Notes:

• You must ensure that the default Docker runtime is runc to make use of the USE_DOCKER variable. If that is not the case, remove the variable from the previous command. See Checking Docker default runtime.
• If you do not wish to build under Docker, remove the USE_DOCKER variable in the previous command and ensure the qemu-img command is available on your system.

Install the rootfs image

$ commit=$(git log --format=%h -1 HEAD)
$ date=$(date +%Y-%m-%d-%T.%N%z)
$ image="kata-containers-${date}-${commit}"
$ sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}"
$ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img)

Create an initrd image - OPTIONAL

Create a local rootfs for initrd image

$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs"
$ sudo rm -rf ${ROOTFS_DIR}
$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
$ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh ${distro}'

AGENT_INIT controls if the guest image uses kata agent as the guest init process. When you create an initrd image, always set AGENT_INIT to yes.

You MUST choose one of alpine, centos, clearlinux, euleros, and fedora for ${distro}.

Optionally, add your custom agent binary to the rootfs with the following:

$ sudo install -o root -g root -m 0550 -T ../../agent/kata-agent ${ROOTFS_DIR}/sbin/init

Build an initrd image

$ cd $GOPATH/src/github.com/kata-containers/osbuilder/initrd-builder
$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}'

Install the initrd image

$ commit=$(git log --format=%h -1 HEAD)
$ date=$(date +%Y-%m-%d-%T.%N%z)
$ image="kata-containers-initrd-${date}-${commit}"
$ sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}"
$ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img)

Install guest kernel images

As a prerequisite, you need to install libelf-dev and bc. Otherwise, you will not be able to build the kernel from sources.

$ kernel_arch="$(arch)"
$ tmpdir="$(mktemp -d)"
$ pushd "$tmpdir"
$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/configs/x86_kata_kvm_4.14.x -o .config
$ kernel_version=$(grep "Linux/[${kernel_arch}]*" .config | cut -d' ' -f3 | tail -1)
$ kernel_tar_file="linux-${kernel_version}.tar.xz"
$ kernel_url="https://cdn.kernel.org/pub/linux/kernel/v$(echo $kernel_version | cut -f1 -d.).x/${kernel_tar_file}"
$ curl -LOk ${kernel_url}
$ tar -xf ${kernel_tar_file}
$ mv .config "linux-${kernel_version}"
$ pushd "linux-${kernel_version}"
$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/patches/0001-NO-UPSTREAM-9P-always-use-cached-inode-to-fill-in-v9.patch | patch -p1
$ make ARCH=${kernel_arch} -j$(nproc)
$ kata_kernel_dir="/usr/share/kata-containers"
$ kata_vmlinuz="${kata_kernel_dir}/kata-vmlinuz-${kernel_version}.container"
$ sudo install -o root -g root -m 0755 -D "$(realpath arch/${kernel_arch}/boot/bzImage)" "${kata_vmlinuz}"
$ sudo ln -sf "${kata_vmlinuz}" "${kata_kernel_dir}/vmlinuz.container"
$ kata_vmlinux="${kata_kernel_dir}/kata-vmlinux-${kernel_version}"
$ sudo install -o root -g root -m 0755 -D "$(realpath vmlinux)" "${kata_vmlinux}"
$ sudo ln -sf "${kata_vmlinux}" "${kata_kernel_dir}/vmlinux.container"
$ popd
$ popd
$ rm -rf "${tmpdir}"

Run Kata Containers with Docker

Update Docker configuration

$ dir=/etc/systemd/system/docker.service.d
$ file="$dir/kata-containers.conf"
$ sudo mkdir -p "$dir"
$ sudo test -e "$file" || echo -e "[Service]\nType=simple\nExecStart=\nExecStart=/usr/bin/dockerd -D --default-runtime runc" | sudo tee "$file"
$ sudo grep -q "kata-runtime=" $file || sudo sed -i 's!^\(ExecStart=[^$].*$\)!\1 --add-runtime kata-runtime=/usr/local/bin/kata-runtime!g' "$file"
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

Create a container using Kata

$ sudo docker run -ti --runtime kata-runtime busybox sh

Troubleshoot Kata Containers

If you are unable to create a Kata Container first ensure you have enabled full debug before attempting to create a container. Then run the kata-collect-data.sh script and paste its output directly into a github issue.

Note:

The kata-collect-data.sh script is built from the runtime repository.

To perform analysis on Kata logs, use the kata-log-parser tool, which can convert the logs into formats (e.g. JSON, TOML, XML, and YAML).

To obtain a full backtrace for the agent, proxy, runtime, or shim send the SIGUSR1 signal to the process ID of the component. The component will send a backtrace to the system log on the host system and continue to run without interruption.

For example, to obtain a backtrace for kata-proxy:

$ sudo kill -USR1 $kata_proxy_pid
$ sudo journalctl -t kata-proxy

See Set up a debug console.

Appendices

Checking Docker default runtime

$ sudo docker info 2>/dev/null | grep -i "default runtime" | cut -d: -f2- | grep -q runc  && echo "SUCCESS" || echo "ERROR: Incorrect default Docker runtime"

Set up a debug console

By default you cannot login to a virtual machine since this can be sensitive from a security perspective. Also allowing logins would require additional packages in the rootfs, which would increase the size of the image used to boot the virtual machine.

If you want to login to a virtual machine that hosts your containers, complete the following steps, which assume a rootfs image.

Create a custom image containing a shell

To login to a virtual machine, you must create a custom rootfs containing a shell such as bash(1).

For example using CentOS:

$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true EXTRA_PKGS="bash" ./rootfs.sh centos'

Create a debug systemd service

Create the service file that starts the shell in the rootfs directory:

$ cat <<EOT | sudo tee ${ROOTFS_DIR}/lib/systemd/system/kata-debug.service
[Unit]
Description=Kata Containers debug console

[Service]
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
StandardInput=tty
StandardOutput=tty
PrivateDevices=yes
Type=simple
ExecStart=/usr/bin/bash
Restart=always
EOT

Note: You might need to adjust the ExecStart= path.

Add a dependency to start the debug console:

$ sudo sed -i '$a Requires=kata-debug.service' ${ROOTFS_DIR}/lib/systemd/system/kata-containers.target

Build the debug image

Follow the instructions in the Build a rootfs image section.

Configure runtime for custom debug image

Install the image:

$ name="kata-containers-centos-with-debug-console.img"
$ sudo install -o root -g root -m 0640 kata-containers.img "/usr/share/kata-containers/${name}"

Next, modify the image= values in the [hypervisor.qemu] section of the configuration file to specify the full path to the image name specified in the previous code section. Alternatively, recreate the symbolic link so it points to the new debug image:

$ (cd /usr/share/kata-containers && sudo ln -sf "$name" kata-containers.img)

Note: You should take care to undo this change after you finish debugging to avoid all subsequently created containers from using the debug image.

Ensure debug options are valid

For the debug console to work, you must ensure that proxy debug is disabled in the configuration file. If proxy debug is enabled, you will not see any output when you connect to the virtual machine:

$ sudo awk '{if (/^\[proxy\.kata\]/) {got=1}; if (got == 1 && /^.*enable_debug/) {print "#enable_debug = true"; got=0; next; } else {print}}' /usr/share/defaults/kata-containers/configuration.toml > /tmp/configuration.toml
$ sudo install -o root -g root -m 0640 /tmp/configuration.toml /usr/share/defaults/kata-containers/configuration.toml

Create a container

Create a container as normal. For example using Docker:

$ sudo docker run -ti busybox sh

Connect to the virtual machine using the debug console

$ id=$(sudo docker ps -q --no-trunc)
$ console="/var/run/vc/sbs/${id}/console.sock"
$ sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}"

Note: You need to press the RETURN key to see the shell prompt.

To disconnect from the virtual machine, type CONTROL+q (hold down the CONTROL key and press q).

Obtain details of the image

If the image is created using osbuilder, the following YAML file exists and contains details of the image and how it was created:

$ cat /var/lib/osbuilder/osbuilder.yaml