diff --git a/.github/workflows/kata-deploy-test.yaml b/.github/workflows/kata-deploy-test.yaml index 4a9c06de1..e0d6afd7c 100644 --- a/.github/workflows/kata-deploy-test.yaml +++ b/.github/workflows/kata-deploy-test.yaml @@ -1,4 +1,5 @@ on: + workflow_dispatch: # this is used to trigger the workflow on non-main branches issue_comment: types: [created, edited] diff --git a/README.md b/README.md index 3cac62400..90a5c9209 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,7 @@ The table below lists the remaining parts of the project: Kata Containers is now [available natively for most distributions](docs/install/README.md#packaged-installation-methods). -However, packaging scripts and metadata are still used to generate snap and GitHub releases. See +However, packaging scripts and metadata are still used to generate [snap](snap/local) and GitHub releases. See the [components](#components) section for further details. ## Glossary of Terms diff --git a/docs/Release-Process.md b/docs/Release-Process.md index 9dfc37041..7dcfb84a3 100644 --- a/docs/Release-Process.md +++ b/docs/Release-Process.md @@ -4,11 +4,11 @@ ## Requirements - [hub](https://github.com/github/hub) - * Using an [application token](https://github.com/settings/tokens) is required for hub. + * Using an [application token](https://github.com/settings/tokens) is required for hub (set to a GITHUB_TOKEN environment variable). - GitHub permissions to push tags and create releases in Kata repositories. -- GPG configured to sign git tags. https://help.github.com/articles/generating-a-new-gpg-key/ +- GPG configured to sign git tags. https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key - You should configure your GitHub to use your ssh keys (to push to branches). See https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/. * As an alternative, configure hub to push and fork with HTTPS, `git config --global hub.protocol https` (Not tested yet) * @@ -48,7 +48,7 @@ ### Merge all bump version Pull requests - The above step will create a GitHub pull request in the Kata projects. Trigger the CI using `/test` command on each bump Pull request. - - Trigger the test-kata-deploy workflow on the kata-containers repository bump Pull request using `/test_kata_deploy` (monitor under the "action" tab). + - Trigger the `test-kata-deploy` workflow which is under the `Actions` tab on the repository GitHub page (make sure to select the correct branch and validate it passes). - Check any failures and fix if needed. - Work with the Kata approvers to verify that the CI works and the pull requests are merged. diff --git a/docs/design/architecture/storage.md b/docs/design/architecture/storage.md index 974f260c3..8cdda1e2e 100644 --- a/docs/design/architecture/storage.md +++ b/docs/design/architecture/storage.md @@ -20,7 +20,7 @@ For virtio-fs, the [runtime](README.md#runtime) starts one `virtiofsd` daemon ## Devicemapper The -[devicemapper `snapshotter`](https://github.com/containerd/containerd/tree/master/snapshots/devmapper) +[devicemapper `snapshotter`](https://github.com/containerd/containerd/tree/main/snapshots/devmapper) is a special case. The `snapshotter` uses dedicated block devices rather than formatted filesystems, and operates at the block level rather than the file level. This knowledge is used to directly use the diff --git a/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md b/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md index 32943b8d3..18a75bde7 100644 --- a/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md +++ b/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md @@ -2,20 +2,20 @@ An NVIDIA GPU device can be passed to a Kata Containers container using GPU passthrough (NVIDIA GPU pass-through mode) as well as GPU mediated passthrough -(NVIDIA vGPU mode). +(NVIDIA `vGPU` mode). NVIDIA GPU pass-through mode, an entire physical GPU is directly assigned to one VM, bypassing the NVIDIA Virtual GPU Manager. In this mode of operation, the GPU is accessed exclusively by the NVIDIA driver running in the VM to which it is assigned. The GPU is not shared among VMs. -NVIDIA Virtual GPU (vGPU) enables multiple virtual machines (VMs) to have +NVIDIA Virtual GPU (`vGPU`) enables multiple virtual machines (VMs) to have simultaneous, direct access to a single physical GPU, using the same NVIDIA graphics drivers that are deployed on non-virtualized operating systems. By -doing this, NVIDIA vGPU provides VMs with unparalleled graphics performance, +doing this, NVIDIA `vGPU` provides VMs with unparalleled graphics performance, compute performance, and application compatibility, together with the cost-effectiveness and scalability brought about by sharing a GPU among multiple -workloads. A vGPU can be either time-sliced or Multi-Instance GPU (MIG)-backed +workloads. A `vGPU` can be either time-sliced or Multi-Instance GPU (MIG)-backed with [MIG-slices](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/). | Technology | Description | Behavior | Detail | @@ -46,14 +46,14 @@ $ lspci -s d0:00.0 -vv | grep Region For large BARs devices, MMIO mapping above 4G address space should be `enabled` in the PCI configuration of the BIOS. -Some hardware vendors use different name in BIOS, such as: +Some hardware vendors use a different name in BIOS, such as: - Above 4G Decoding - Memory Hole for PCI MMIO - Memory Mapped I/O above 4GB If one is using a GPU based on the Ampere architecture and later additionally -SR-IOV needs to be enabled for the vGPU use-case. +SR-IOV needs to be enabled for the `vGPU` use-case. The following steps outline the workflow for using an NVIDIA GPU with Kata. @@ -154,7 +154,7 @@ $ ./build-kernel.sh -v 5.15.23 -g nvidia build $ sudo -E ./build-kernel.sh -v 5.15.23 -g nvidia install ``` -To build NVIDIA Driver in Kata container, `linux-headers` is required. +To build NVIDIA Driver in Kata container, `linux-headers` are required. This is a way to generate deb packages for `linux-headers`: > **Note**: @@ -177,7 +177,7 @@ kernel = "/usr/share/kata-containers/vmlinuz-nvidia-gpu.container" Use the following steps to pass an NVIDIA GPU device in pass-through mode with Kata: -1. Find the Bus-Device-Function (BDF) for GPU device on host: +1. Find the Bus-Device-Function (BDF) for the GPU device on the host: ```sh $ sudo lspci -nn -D | grep -i nvidia @@ -219,7 +219,7 @@ Use the following steps to pass an NVIDIA GPU device in pass-through mode with K crw-rw-rw- 1 root root 10, 196 Mar 18 02:27 vfio ``` -4. Start a Kata container with GPU device: +4. Start a Kata container with the GPU device: ```sh # You may need to `modprobe vhost-vsock` if you get @@ -246,9 +246,228 @@ Use the following steps to pass an NVIDIA GPU device in pass-through mode with K ## NVIDIA vGPU mode with Kata Containers NVIDIA vGPU is a licensed product on all supported GPU boards. A software license -is required to enable all vGPU features within the guest VM. +is required to enable all vGPU features within the guest VM. NVIDIA vGPU manager +needs to be installed on the host to configure GPUs in vGPU mode. See [NVIDIA Virtual GPU Software Documentation v14.0 through 14.1](https://docs.nvidia.com/grid/14.0/) for more details. -> **TODO**: Will follow up with instructions +### NVIDIA vGPU time-sliced + +In the time-sliced mode, the GPU is not partitioned and the workload uses the +whole GPU and shares access to the GPU engines. Processes are scheduled in +series. The best effort scheduler is the default one and can be exchanged by +other scheduling policies see the documentation above how to do that. + +Beware if you had `MIG` enabled before to disable `MIG` on the GPU if you want +to use `time-sliced` `vGPU`. + +```sh +$ sudo nvidia-smi -mig 0 +``` + +Enable the virtual functions for the physical GPU in the `sysfs` file system. + +```sh +$ sudo /usr/lib/nvidia/sriov-manage -e 0000:41:00.0 +``` + +Get the `BDF` of the available virtual function on the GPU, and choose one for the +following steps. + +```sh +$ cd /sys/bus/pci/devices/0000:41:00.0/ +$ ls -l | grep virtfn +``` + +#### List all available vGPU instances + +The following shell snippet will walk the `sysfs` and only print instances +that are available, that can be created. + +```sh +# The 00.0 is often the PF of the device the VFs will have the funciont in the +# BDF incremented by some values so e.g. the very first VF is 0000:41:00.4 + +cd /sys/bus/pci/devices/0000:41:00.0/ + +for vf in $(ls -d virtfn*) +do + BDF=$(basename $(readlink -f $vf)) + for md in $(ls -d $vf/mdev_supported_types/*) + do + AVAIL=$(cat $md/available_instances) + NAME=$(cat $md/name) + DIR=$(basename $md) + + if [ $AVAIL -gt 0 ]; then + echo "| BDF | INSTANCES | NAME | DIR |" + echo "+--------------+-----------+----------------+------------+" + printf "| %12s |%10d |%15s | %10s |\n\n" "$BDF" "$AVAIL" "$NAME" "$DIR" + fi + + done +done +``` + +If there are available instances you get something like this (for the first VF), +beware that the output is highly dependent on the GPU you have, if there is no +output check again if `MIG` is really disabled. + +```sh +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-4C | nvidia-692 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-8C | nvidia-693 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-10C | nvidia-694 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-16C | nvidia-695 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-20C | nvidia-696 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-40C | nvidia-697 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-80C | nvidia-698 | + +``` + +Change to the `mdev_supported_types` directory for the virtual function on which +you want to create the `vGPU`. Taking the first output as an example: + +```sh +$ cd virtfn0/mdev_supported_types/nvidia-692 +$ UUIDGEN=$(uuidgen) +$ sudo bash -c "echo $UUIDGEN > create" +``` + +Confirm that the `vGPU` was created. You should see the `UUID` pointing to a +subdirectory of the `sysfs` space. + +```sh +$ ls -l /sys/bus/mdev/devices/ +``` + +Get the `IOMMU` group number and verify there is a `VFIO` device created to use +with Kata. + +```sh +$ ls -l /sys/bus/mdev/devices/*/ +$ ls -l /dev/vfio +``` + +Use the `VFIO` device created in the same way as in the pass-through use-case. +Beware that the guest needs the NVIDIA guest drivers, so one would need to build +a new guest `OS` image. + +### NVIDIA vGPU MIG-backed + +We're not going into detail what `MIG` is but briefly it is a technology to +partition the hardware into independent instances with guaranteed quality of +service. For more details see [NVIDIA Multi-Instance GPU User Guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/). + +First enable `MIG` mode for a GPU, depending on the platform you're running +a reboot would be necessary. Some platforms support GPU reset. + +```sh +$ sudo nvidia-smi -mig 1 +``` + +If the platform supports a GPU reset one can run, otherwise you will get a +warning to reboot the server. + +```sh +$ sudo nvidia-smi --gpu-reset +``` + +The driver per default provides a number of profiles that users can opt-in when +configuring the MIG feature. + +```sh +$ sudo nvidia-smi mig -lgip ++-----------------------------------------------------------------------------+ +| GPU instance profiles: | +| GPU Name ID Instances Memory P2P SM DEC ENC | +| Free/Total GiB CE JPEG OFA | +|=============================================================================| +| 0 MIG 1g.10gb 19 7/7 9.50 No 14 0 0 | +| 1 0 0 | ++-----------------------------------------------------------------------------+ +| 0 MIG 1g.10gb+me 20 1/1 9.50 No 14 1 0 | +| 1 1 1 | ++-----------------------------------------------------------------------------+ +| 0 MIG 2g.20gb 14 3/3 19.50 No 28 1 0 | +| 2 0 0 | ++-----------------------------------------------------------------------------+ + ... +``` + +Create the GPU instances that correspond to the `vGPU` types of the `MIG-backed` +`vGPUs` that you will create [NVIDIA A100 PCIe 80GB Virtual GPU Types](https://docs.nvidia.com/grid/13.0/grid-vgpu-user-guide/index.html#vgpu-types-nvidia-a100-pcie-80gb). + +```sh +# MIG 1g.10gb --> vGPU A100D-1-10C +$ sudo nvidia-smi mig -cgi 19 +``` + +List the GPU instances and get the GPU instance id to create the compute +instance. + +```sh +$ sudo nvidia-smi mig -lgi # list the created GPU instances +$ sudo nvidia-smi mig -cci -gi 9 # each GPU instance can have several compute + # instances. Instance -> Workload +``` + +Verify that the compute instances were created within the GPU instance + +```sh +$ nvidia-smi + ... snip ... ++-----------------------------------------------------------------------------+ +| MIG devices: | ++------------------+----------------------+-----------+-----------------------+ +| GPU GI CI MIG | Memory-Usage | Vol| Shared | +| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG| +| | | ECC| | +|==================+======================+===========+=======================| +| 0 9 0 0 | 0MiB / 9728MiB | 14 0 | 1 0 0 0 0 | +| | 0MiB / 4095MiB | | | ++------------------+----------------------+-----------+-----------------------+ + ... snip ... +``` + +We can use the [snippet](#list-all-available-vgpu-instances) from before to list +the available `vGPU` instances, this time `MIG-backed`. + +```sh +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 |GRID A100D-1-10C | nvidia-699 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.5 | 1 |GRID A100D-1-10C | nvidia-699 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:01.6 | 1 |GRID A100D-1-10C | nvidia-699 | + ... snip ... +``` + +Repeat the steps after the [snippet](#list-all-available-vgpu-instances) listing +to create the corresponding `mdev` device and use the guest `OS` created in the +previous section with `time-sliced` `vGPUs`. ## Install NVIDIA Driver + Toolkit in Kata Containers Guest OS @@ -263,7 +482,7 @@ export EXTRA_PKGS="gcc make curl gnupg" ``` Having the `$ROOTFS_DIR` exported in the previous step we can now install all the -need parts in the guest OS. In this case we have an Ubuntu based rootfs. +needed parts in the guest OS. In this case, we have an Ubuntu based rootfs. First off all mount the special filesystems into the rootfs @@ -281,9 +500,9 @@ Now we can enter `chroot` $ sudo chroot ${ROOTFS_DIR} ``` -Inside the rootfs one is going to install the drivers and toolkit to enable easy -creation of GPU containers with Kata. We can also use this rootfs for any other -container not specifically only for GPUs. +Inside the rootfs one is going to install the drivers and toolkit to enable the +easy creation of GPU containers with Kata. We can also use this rootfs for any +other container not specifically only for GPUs. As a prerequisite install the copied kernel development packages @@ -304,6 +523,7 @@ $ ./NVIDIA-Linux-x86_64-510.54.run -x $ cd NVIDIA-Linux-x86_64-510.54 $ ./nvidia-installer -k 5.15.23-nvidia-gpu ``` + Having the drivers installed we need to install the toolkit which will take care of providing the right bits into the container. @@ -325,7 +545,7 @@ Create the hook execution file for Kata: /usr/bin/nvidia-container-toolkit -debug $@ ``` -As a last step one can do some cleanup of files or package caches. Build the +As the last step one can do some cleanup of files or package caches. Build the rootfs and configure it for use with Kata according to the development guide. Enable the `guest_hook_path` in Kata's `configuration.toml` @@ -334,7 +554,7 @@ Enable the `guest_hook_path` in Kata's `configuration.toml` guest_hook_path = "/usr/share/oci/hooks" ``` -One has build a NVIDIA rootfs, kernel and now we can run any GPU container +One has built a NVIDIA rootfs, kernel and now we can run any GPU container without installing the drivers into the container. Check NVIDIA device status with `nvidia-smi` @@ -362,7 +582,7 @@ Fri Mar 18 10:36:59 2022 +-----------------------------------------------------------------------------+ ``` -As a last step one can remove the additional packages and files that were added +As the last step one can remove the additional packages and files that were added to the `$ROOTFS_DIR` to keep it as small as possible. ## References diff --git a/docs/use-cases/using-Intel-QAT-and-kata.md b/docs/use-cases/using-Intel-QAT-and-kata.md index 3a7e6cf68..0bcd2e3bb 100644 --- a/docs/use-cases/using-Intel-QAT-and-kata.md +++ b/docs/use-cases/using-Intel-QAT-and-kata.md @@ -312,7 +312,7 @@ working properly with the Kata Containers VM. ### Build OpenSSL Intel® QAT engine container -Use the OpenSSL Intel® QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/master/demo/openssl-qat-engine) +Use the OpenSSL Intel® QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/main/demo/openssl-qat-engine) to build a container image with an optimized OpenSSL engine for Intel® QAT. Using `docker build` with the Kata Containers runtime can sometimes have issues. Therefore, make sure that `runc` is the default Docker container @@ -444,7 +444,7 @@ $ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest $ sudo ctr -n=k8s.io images import openssl-qat-engine.tar ``` -The [Intel® QAT Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/cmd/qat_plugin/README.md) +The [Intel® QAT Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/qat_plugin/README.md) needs to be started so that the virtual functions can be discovered and used by Kubernetes. diff --git a/snap/README.md b/snap/local/README.md similarity index 64% rename from snap/README.md rename to snap/local/README.md index 3729542ec..4b449ef17 100644 --- a/snap/README.md +++ b/snap/local/README.md @@ -22,21 +22,35 @@ $ sudo snap install kata-containers --classic ## Build and install snap image -Run next command at the root directory of the packaging repository. +Run the command below which will use the packaging Makefile to build the snap image: ```sh -$ make snap +$ make -C tools/packaging snap ``` +> **Warning:** +> +> By default, `snapcraft` will create a clean virtual machine +> environment to build the snap in using the `multipass` tool. +> +> However, `multipass` is silently disabled when `--destructive-mode` is +> used. +> +> Since building the Kata Containers package currently requires +> `--destructive-mode`, the snap will be built using the host +> environment. To avoid parts of the build auto-detecting additional +> features to enable (for example for QEMU), we recommend that you +> only run the snap build in a minimal host environment. + To install the resulting snap image, snap must be put in [classic mode][3] and the -security confinement must be disabled (*--classic*). Also since the resulting snap -has not been signed the verification of signature must be omitted (*--dangerous*). +security confinement must be disabled (`--classic`). Also since the resulting snap +has not been signed the verification of signature must be omitted (`--dangerous`). ```sh -$ sudo snap install --classic --dangerous kata-containers_[VERSION]_[ARCH].snap +$ sudo snap install --classic --dangerous "kata-containers_${version}_${arch}.snap" ``` -Replace `VERSION` with the current version of Kata Containers and `ARCH` with +Replace `${version}` with the current version of Kata Containers and `${arch}` with the system architecture. ## Configure Kata Containers @@ -76,12 +90,12 @@ then a new configuration file can be [created](#configure-kata-containers) and [configured][7]. [1]: https://docs.snapcraft.io/snaps/intro -[2]: ../docs/design/architecture/README.md#root-filesystem-image +[2]: ../../docs/design/architecture/README.md#root-filesystem-image [3]: https://docs.snapcraft.io/reference/confinement#classic -[4]: https://github.com/kata-containers/runtime#configuration +[4]: https://github.com/kata-containers/kata-containers/tree/main/src/runtime#configuration [5]: https://docs.docker.com/engine/reference/commandline/dockerd -[6]: ../docs/install/docker/ubuntu-docker-install.md -[7]: ../docs/Developer-Guide.md#configure-to-use-initrd-or-rootfs-image +[6]: ../../docs/install/docker/ubuntu-docker-install.md +[7]: ../../docs/Developer-Guide.md#configure-to-use-initrd-or-rootfs-image [8]: https://snapcraft.io/kata-containers -[9]: ../docs/Developer-Guide.md#run-kata-containers-with-docker -[10]: ../docs/Developer-Guide.md#run-kata-containers-with-kubernetes +[9]: ../../docs/Developer-Guide.md#run-kata-containers-with-docker +[10]: ../../docs/Developer-Guide.md#run-kata-containers-with-kubernetes diff --git a/snap/local/snap-common.sh b/snap/local/snap-common.sh new file mode 100644 index 000000000..0a2a18e15 --- /dev/null +++ b/snap/local/snap-common.sh @@ -0,0 +1,114 @@ +#!/usr/bin/env bash +# +# Copyright (c) 2022 Intel Corporation +# +# SPDX-License-Identifier: Apache-2.0 + +# Description: Idempotent script to be sourced by all parts in a +# snapcraft config file. + +set -o errexit +set -o nounset +set -o pipefail + +# XXX: Bash-specific code. zsh doesn't support this option and that *does* +# matter if this script is run sourced... since it'll be using zsh! ;) +[ -n "$BASH_VERSION" ] && set -o errtrace + +[ -n "${DEBUG:-}" ] && set -o xtrace + +die() +{ + echo >&2 "ERROR: $0: $*" +} + +[ -n "${SNAPCRAFT_STAGE:-}" ] ||\ + die "must be sourced from a snapcraft config file" + +snap_yq_version=3.4.1 + +snap_common_install_yq() +{ + export yq="${SNAPCRAFT_STAGE}/bin/yq" + + local yq_pkg + yq_pkg="github.com/mikefarah/yq" + + local yq_url + yq_url="https://${yq_pkg}/releases/download/${snap_yq_version}/yq_${goos}_${goarch}" + curl -o "${yq}" -L "${yq_url}" + chmod +x "${yq}" +} + +# Function that should be called for each snap "part" in +# snapcraft.yaml. +snap_common_main() +{ + # Architecture + arch="$(uname -m)" + + case "${arch}" in + aarch64) + goarch="arm64" + qemu_arch="${arch}" + ;; + + ppc64le) + goarch="ppc64le" + qemu_arch="ppc64" + ;; + + s390x) + goarch="${arch}" + qemu_arch="${arch}" + ;; + + x86_64) + goarch="amd64" + qemu_arch="${arch}" + ;; + + *) die "unsupported architecture: ${arch}" ;; + esac + + dpkg_arch=$(dpkg --print-architecture) + + # golang + # + # We need the O/S name in golang format, but since we don't + # know if the godeps part has run, we don't know if golang is + # available yet, hence fall back to a standard system command. + goos="$(go env GOOS &>/dev/null || true)" + [ -z "$goos" ] && goos=$(uname -s|tr '[A-Z]' '[a-z]') + + export GOROOT="${SNAPCRAFT_STAGE}" + export GOPATH="${GOROOT}/gopath" + export GO111MODULE="auto" + + mkdir -p "${GOPATH}/bin" + export PATH="${GOPATH}/bin:${PATH}" + + # Proxy + export http_proxy="${http_proxy:-}" + export https_proxy="${https_proxy:-}" + + # Binaries + mkdir -p "${SNAPCRAFT_STAGE}/bin" + + export PATH="$PATH:${SNAPCRAFT_STAGE}/bin" + + # YAML query tool + export yq="${SNAPCRAFT_STAGE}/bin/yq" + + # Kata paths + export kata_dir=$(printf "%s/src/github.com/%s/%s" \ + "${GOPATH}" \ + "${SNAPCRAFT_PROJECT_NAME}" \ + "${SNAPCRAFT_PROJECT_NAME}") + + export versions_file="${kata_dir}/versions.yaml" + + [ -n "${yq:-}" ] && [ -x "${yq:-}" ] || snap_common_install_yq +} + +snap_common_main diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml index 94fd2a8c9..badfc99ee 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -19,6 +19,8 @@ parts: - git - git-extras override-pull: | + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + version="9999" if echo "${GITHUB_REF:-}" | grep -q -E "^refs/tags"; then @@ -29,9 +31,6 @@ parts: snapcraftctl set-grade "stable" snapcraftctl set-version "${version}" - # setup GOPATH - this repo dir should be there - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} mkdir -p $(dirname ${kata_dir}) ln -sf $(realpath "${SNAPCRAFT_STAGE}/..") ${kata_dir} @@ -43,28 +42,12 @@ parts: build-packages: - curl override-build: | + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + # put everything in stage - cd ${SNAPCRAFT_STAGE} + cd "${SNAPCRAFT_STAGE}" - mkdir -p "${SNAPCRAFT_STAGE}/bin/" - yq_path="${SNAPCRAFT_STAGE}/bin/yq" - yq_pkg="github.com/mikefarah/yq" - goos="linux" - case "$(uname -m)" in - aarch64) goarch="arm64";; - ppc64le) goarch="ppc64le";; - x86_64) goarch="amd64";; - s390x) goarch="s390x";; - *) echo "unsupported architecture: $(uname -m)"; exit 1;; - esac - - yq_version=3.4.1 - yq_url="https://${yq_pkg}/releases/download/${yq_version}/yq_${goos}_${goarch}" - curl -o "${yq_path}" -L "${yq_url}" - chmod +x "${yq_path}" - - kata_dir=gopath/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - version="$(${yq_path} r ${kata_dir}/versions.yaml languages.golang.meta.newest-version)" + version="$(${yq} r ${kata_dir}/versions.yaml languages.golang.meta.newest-version)" tarfile="go${version}.${goos}-${goarch}.tar.gz" curl -LO https://golang.org/dl/${tarfile} tar -xf ${tarfile} --strip-components=1 @@ -81,28 +64,17 @@ parts: - uidmap - gnupg2 override-build: | - [ "$(uname -m)" = "ppc64le" ] || [ "$(uname -m)" = "s390x" ] && sudo apt-get --no-install-recommends install -y protobuf-compiler + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" - yq=${SNAPCRAFT_STAGE}/bin/yq + [ "${arch}" = "ppc64le" ] || [ "${arch}" = "s390x" ] && sudo apt-get --no-install-recommends install -y protobuf-compiler - # set GOPATH - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - - export GOROOT=${SNAPCRAFT_STAGE} - export PATH="${GOROOT}/bin:${PATH}" - export GO111MODULE="auto" - - http_proxy=${http_proxy:-""} - https_proxy=${https_proxy:-""} if [ -n "$http_proxy" ]; then echo "Setting proxy $http_proxy" - sudo -E systemctl set-environment http_proxy=$http_proxy || true - sudo -E systemctl set-environment https_proxy=$https_proxy || true + sudo -E systemctl set-environment http_proxy="$http_proxy" || true + sudo -E systemctl set-environment https_proxy="$https_proxy" || true fi # Copy yq binary. It's used in the container - mkdir -p "${GOPATH}/bin/" cp -a "${yq}" "${GOPATH}/bin/" echo "Unmasking docker service" @@ -113,63 +85,54 @@ parts: echo "Starting docker" sudo -E systemctl start docker || true - cd ${kata_dir}/tools/osbuilder + cd "${kata_dir}/tools/osbuilder" # build image export AGENT_INIT=yes export USE_DOCKER=1 export DEBUG=1 - arch="$(uname -m)" initrd_distro=$(${yq} r -X ${kata_dir}/versions.yaml assets.initrd.architecture.${arch}.name) image_distro=$(${yq} r -X ${kata_dir}/versions.yaml assets.image.architecture.${arch}.name) case "$arch" in x86_64) # In some build systems it's impossible to build a rootfs image, try with the initrd image - sudo -E PATH=$PATH make image DISTRO=${image_distro} || sudo -E PATH=$PATH make initrd DISTRO=${initrd_distro} + sudo -E PATH=$PATH make image DISTRO="${image_distro}" || sudo -E PATH="$PATH" make initrd DISTRO="${initrd_distro}" ;; aarch64|ppc64le|s390x) - sudo -E PATH=$PATH make initrd DISTRO=${initrd_distro} + sudo -E PATH="$PATH" make initrd DISTRO="${initrd_distro}" ;; - *) echo "unsupported architecture: $(uname -m)"; exit 1;; + *) die "unsupported architecture: ${arch}" ;; esac # Install image - kata_image_dir=${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers - mkdir -p ${kata_image_dir} - cp kata-containers*.img ${kata_image_dir} + kata_image_dir="${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers" + mkdir -p "${kata_image_dir}" + cp kata-containers*.img "${kata_image_dir}" runtime: after: [godeps, image, cloud-hypervisor] plugin: nil build-attributes: [no-patchelf] override-build: | - # set GOPATH - export GOPATH=${SNAPCRAFT_STAGE}/gopath - export GOROOT=${SNAPCRAFT_STAGE} - export PATH="${GOROOT}/bin:${PATH}" - export GO111MODULE="auto" - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" - cd ${kata_dir}/src/runtime + cd "${kata_dir}/src/runtime" - # setup arch - arch=$(uname -m) - if [ ${arch} = "ppc64le" ]; then - arch="ppc64" - fi + qemu_cmd="qemu-system-${qemu_arch}" # build and install runtime make \ - PREFIX=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr \ + PREFIX="/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr" \ SKIP_GO_VERSION_CHECK=1 \ - QEMUCMD=qemu-system-$arch + QEMUCMD="${qemu_cmd}" + make install \ PREFIX=/usr \ - DESTDIR=${SNAPCRAFT_PART_INSTALL} \ + DESTDIR="${SNAPCRAFT_PART_INSTALL}" \ SKIP_GO_VERSION_CHECK=1 \ - QEMUCMD=qemu-system-$arch + QEMUCMD="${qemu_cmd}" if [ ! -f ${SNAPCRAFT_PART_INSTALL}/../../image/install/usr/share/kata-containers/kata-containers.img ]; then sed -i -e "s|^image =.*|initrd = \"/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr/share/kata-containers/kata-containers-initrd.img\"|" \ @@ -186,44 +149,37 @@ parts: - bison - flex override-build: | - yq=${SNAPCRAFT_STAGE}/bin/yq - export PATH="${PATH}:${SNAPCRAFT_STAGE}" - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - versions_file="${kata_dir}/versions.yaml" + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + kernel_version="$(${yq} r $versions_file assets.kernel.version)" #Remove extra 'v' - kernel_version=${kernel_version#v} + kernel_version="${kernel_version#v}" - [ "$(uname -m)" = "s390x" ] && sudo apt-get --no-install-recommends install -y libssl-dev + [ "${arch}" = "s390x" ] && sudo apt-get --no-install-recommends install -y libssl-dev - export GOPATH=${SNAPCRAFT_STAGE}/gopath - export GO111MODULE="auto" - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - - cd ${kata_dir}/tools/packaging/kernel + cd "${kata_dir}/tools/packaging/kernel" kernel_dir_prefix="kata-linux-" # Setup and build kernel - ./build-kernel.sh -v ${kernel_version} -d setup + ./build-kernel.sh -v "${kernel_version}" -d setup cd ${kernel_dir_prefix}* make -j $(($(nproc)-1)) EXTRAVERSION=".container" - kernel_suffix=${kernel_version}.container - kata_kernel_dir=${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers - mkdir -p ${kata_kernel_dir} + kernel_suffix="${kernel_version}.container" + kata_kernel_dir="${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers" + mkdir -p "${kata_kernel_dir}" # Install bz kernel - make install INSTALL_PATH=${kata_kernel_dir} EXTRAVERSION=".container" || true - vmlinuz_name=vmlinuz-${kernel_suffix} - ln -sf ${vmlinuz_name} ${kata_kernel_dir}/vmlinuz.container + make install INSTALL_PATH="${kata_kernel_dir}" EXTRAVERSION=".container" || true + vmlinuz_name="vmlinuz-${kernel_suffix}" + ln -sf "${vmlinuz_name}" "${kata_kernel_dir}/vmlinuz.container" # Install raw kernel - vmlinux_path=vmlinux - [ "$(uname -m)" = "s390x" ] && vmlinux_path=arch/s390/boot/compressed/vmlinux - vmlinux_name=vmlinux-${kernel_suffix} - cp ${vmlinux_path} ${kata_kernel_dir}/${vmlinux_name} - ln -sf ${vmlinux_name} ${kata_kernel_dir}/vmlinux.container + vmlinux_path="vmlinux" + [ "${arch}" = "s390x" ] && vmlinux_path="arch/s390/boot/compressed/vmlinux" + vmlinux_name="vmlinux-${kernel_suffix}" + cp "${vmlinux_path}" "${kata_kernel_dir}/${vmlinux_name}" + ln -sf "${vmlinux_name}" "${kata_kernel_dir}/vmlinux.container" qemu: plugin: make @@ -250,12 +206,8 @@ parts: - libselinux1-dev - ninja-build override-build: | - yq=${SNAPCRAFT_STAGE}/bin/yq - export GOPATH=${SNAPCRAFT_STAGE}/gopath - export GO111MODULE="auto" - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" - versions_file="${kata_dir}/versions.yaml" branch="$(${yq} r ${versions_file} assets.hypervisor.qemu.version)" url="$(${yq} r ${versions_file} assets.hypervisor.qemu.url)" commit="" @@ -263,11 +215,11 @@ parts: patches_version_dir="${kata_dir}/tools/packaging/qemu/patches/tag_patches/${branch}" # download source - qemu_dir=${SNAPCRAFT_STAGE}/qemu + qemu_dir="${SNAPCRAFT_STAGE}/qemu" rm -rf "${qemu_dir}" git clone --depth 1 --branch ${branch} --single-branch ${url} "${qemu_dir}" - cd ${qemu_dir} - [ -z "${commit}" ] || git checkout ${commit} + cd "${qemu_dir}" + [ -z "${commit}" ] || git checkout "${commit}" [ -n "$(ls -A ui/keycodemapdb)" ] || git clone --depth 1 https://github.com/qemu/keycodemapdb ui/keycodemapdb/ [ -n "$(ls -A capstone)" ] || git clone --depth 1 https://github.com/qemu/capstone capstone @@ -278,10 +230,10 @@ parts: ${kata_dir}/tools/packaging/scripts/apply_patches.sh "${patches_version_dir}" # Only x86_64 supports libpmem - [ "$(uname -m)" = "x86_64" ] && sudo apt-get --no-install-recommends install -y apt-utils ca-certificates libpmem-dev + [ "${arch}" = "x86_64" ] && sudo apt-get --no-install-recommends install -y apt-utils ca-certificates libpmem-dev - configure_hypervisor=${kata_dir}/tools/packaging/scripts/configure-hypervisor.sh - chmod +x ${configure_hypervisor} + configure_hypervisor="${kata_dir}/tools/packaging/scripts/configure-hypervisor.sh" + chmod +x "${configure_hypervisor}" # static build. The --prefix, --libdir, --libexecdir, --datadir arguments are # based on PREFIX and set by configure-hypervisor.sh echo "$(PREFIX=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr ${configure_hypervisor} -s kata-qemu) \ @@ -291,17 +243,17 @@ parts: # Copy QEMU configurations (Kconfigs) case "${branch}" in "v5.1.0") - cp -a ${kata_dir}/tools/packaging/qemu/default-configs/* default-configs + cp -a "${kata_dir}"/tools/packaging/qemu/default-configs/* default-configs ;; *) - cp -a ${kata_dir}/tools/packaging/qemu/default-configs/* configs/devices/ + cp -a "${kata_dir}"/tools/packaging/qemu/default-configs/* configs/devices/ ;; esac # build and install make -j $(($(nproc)-1)) - make install DESTDIR=${SNAPCRAFT_PART_INSTALL} + make install DESTDIR="${SNAPCRAFT_PART_INSTALL}" prime: - -snap/ - -usr/bin/qemu-ga @@ -321,11 +273,13 @@ parts: plugin: nil after: [godeps] override-build: | + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + # Currently, only one platform uses the new rust virtiofsd. The # others make use of QEMU's C implementation. # # See "tools/packaging/scripts/configure-hypervisor.sh". - if [ "$(uname -m)" = 'x86_64' ] + if [ "${arch}" = 'x86_64' ] then echo "INFO: Building rust version of virtiofsd" else @@ -334,14 +288,8 @@ parts: exit 0 fi - # put everything in stage - cd ${SNAPCRAFT_STAGE} - - export PATH="$PATH:${SNAPCRAFT_STAGE}/bin" - export GOPATH=${SNAPCRAFT_STAGE}/gopath - - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} cd "${kata_dir}" + # Download the rust implementation of virtiofsd tools/packaging/static-build/virtiofsd/build-static-virtiofsd.sh sudo install \ @@ -356,22 +304,31 @@ parts: plugin: nil after: [godeps] override-build: | - arch=$(uname -m) - if [ "{$arch}" == "aarch64" ] || [ "${arch}" == "x64_64" ]; then + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + + if [ "${arch}" == "aarch64" ] || [ "${arch}" == "x86_64" ]; then sudo apt-get -y update sudo apt-get -y install ca-certificates curl gnupg lsb-release - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg - echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + curl -fsSL https://download.docker.com/linux/ubuntu/gpg |\ + sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg + distro_codename=$(lsb_release -cs) + echo "deb [arch=${dpkg_arch} signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu ${distro_codename} stable" |\ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get -y update sudo apt-get -y install docker-ce docker-ce-cli containerd.io sudo systemctl start docker.socket - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - cd ${kata_dir} + cd "${SNAPCRAFT_PROJECT_DIR}" sudo -E NO_TTY=true make cloud-hypervisor-tarball - tar xvJpf build/kata-static-cloud-hypervisor.tar.xz -C /tmp/ - install -D /tmp/opt/kata/bin/cloud-hypervisor ${SNAPCRAFT_PART_INSTALL}/usr/bin/cloud-hypervisor + + tarfile="${SNAPCRAFT_PROJECT_DIR}/tools/packaging/kata-deploy/local-build/build/kata-static-cloud-hypervisor.tar.xz" + tmpdir=$(mktemp -d) + + tar -xvJpf "${tarfile}" -C "${tmpdir}" + + install -D "${tmpdir}/opt/kata/bin/cloud-hypervisor" "${SNAPCRAFT_PART_INSTALL}/usr/bin/cloud-hypervisor" + + rm -rf "${tmpdir}" fi apps: diff --git a/src/agent/rustjail/src/console.rs b/src/agent/rustjail/src/console.rs index 23cf6e840..52e33f392 100644 --- a/src/agent/rustjail/src/console.rs +++ b/src/agent/rustjail/src/console.rs @@ -58,10 +58,7 @@ pub fn setup_master_console(socket_fd: RawFd) -> Result<()> { #[cfg(test)] mod tests { use super::*; - use crate::skip_if_not_root; - use std::fs::File; use std::os::unix::net::UnixListener; - use std::path::PathBuf; use tempfile::{self, tempdir}; const CONSOLE_SOCKET: &str = "console-socket"; diff --git a/src/agent/rustjail/src/container.rs b/src/agent/rustjail/src/container.rs index 88c058868..5f1aa5606 100644 --- a/src/agent/rustjail/src/container.rs +++ b/src/agent/rustjail/src/container.rs @@ -42,7 +42,7 @@ use nix::pty; use nix::sched::{self, CloneFlags}; use nix::sys::signal::{self, Signal}; use nix::sys::stat::{self, Mode}; -use nix::unistd::{self, fork, ForkResult, Gid, Pid, Uid}; +use nix::unistd::{self, fork, ForkResult, Gid, Pid, Uid, User}; use std::os::unix::fs::MetadataExt; use std::os::unix::io::AsRawFd; @@ -64,8 +64,6 @@ use rlimit::{setrlimit, Resource, Rlim}; use tokio::io::AsyncBufReadExt; use tokio::sync::Mutex; -use crate::utils; - pub const EXEC_FIFO_FILENAME: &str = "exec.fifo"; const INIT: &str = "INIT"; @@ -662,12 +660,17 @@ fn do_init_child(cwfd: RawFd) -> Result<()> { } } - // set the "HOME" env getting from "/etc/passwd", if - // there's no uid entry in /etc/passwd, set "/" as the - // home env. if env::var_os(HOME_ENV_KEY).is_none() { - let home_dir = utils::home_dir(guser.uid).unwrap_or_else(|_| String::from("/")); - env::set_var(HOME_ENV_KEY, home_dir); + // try to set "HOME" env by uid + if let Ok(Some(user)) = User::from_uid(Uid::from_raw(guser.uid)) { + if let Ok(user_home_dir) = user.dir.into_os_string().into_string() { + env::set_var(HOME_ENV_KEY, user_home_dir); + } + } + // set default home dir as "/" if "HOME" env is still empty + if env::var_os(HOME_ENV_KEY).is_none() { + env::set_var(HOME_ENV_KEY, String::from("/")); + } } let exec_file = Path::new(&args[0]); @@ -1063,7 +1066,19 @@ impl BaseContainer for LinuxContainer { let st = self.oci_state()?; for pid in self.processes.keys() { - signal::kill(Pid::from_raw(*pid), Some(Signal::SIGKILL))?; + match signal::kill(Pid::from_raw(*pid), Some(Signal::SIGKILL)) { + Err(Errno::ESRCH) => { + info!( + self.logger, + "kill encounters ESRCH, pid: {}, container: {}", + pid, + self.id.clone() + ); + continue; + } + Err(err) => return Err(anyhow!(err)), + Ok(_) => continue, + } } if spec.hooks.is_some() { diff --git a/src/agent/rustjail/src/lib.rs b/src/agent/rustjail/src/lib.rs index d0fe7fe04..aa8be08f1 100644 --- a/src/agent/rustjail/src/lib.rs +++ b/src/agent/rustjail/src/lib.rs @@ -41,7 +41,6 @@ pub mod seccomp; pub mod specconv; pub mod sync; pub mod sync_with_async; -pub mod utils; pub mod validator; use std::collections::HashMap; diff --git a/src/agent/rustjail/src/process.rs b/src/agent/rustjail/src/process.rs index cced9b98f..d94b595cc 100644 --- a/src/agent/rustjail/src/process.rs +++ b/src/agent/rustjail/src/process.rs @@ -5,7 +5,7 @@ use libc::pid_t; use std::fs::File; -use std::os::unix::io::RawFd; +use std::os::unix::io::{AsRawFd, RawFd}; use tokio::sync::mpsc::Sender; use nix::errno::Errno; @@ -137,19 +137,25 @@ impl Process { info!(logger, "before create console socket!"); if !p.tty { - info!(logger, "created console socket!"); + if cfg!(feature = "standard-oci-runtime") { + p.stdin = Some(std::io::stdin().as_raw_fd()); + p.stdout = Some(std::io::stdout().as_raw_fd()); + p.stderr = Some(std::io::stderr().as_raw_fd()); + } else { + info!(logger, "created console socket!"); - let (stdin, pstdin) = unistd::pipe2(OFlag::O_CLOEXEC)?; - p.parent_stdin = Some(pstdin); - p.stdin = Some(stdin); + let (stdin, pstdin) = unistd::pipe2(OFlag::O_CLOEXEC)?; + p.parent_stdin = Some(pstdin); + p.stdin = Some(stdin); - let (pstdout, stdout) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; - p.parent_stdout = Some(pstdout); - p.stdout = Some(stdout); + let (pstdout, stdout) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; + p.parent_stdout = Some(pstdout); + p.stdout = Some(stdout); - let (pstderr, stderr) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; - p.parent_stderr = Some(pstderr); - p.stderr = Some(stderr); + let (pstderr, stderr) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; + p.parent_stderr = Some(pstderr); + p.stderr = Some(stderr); + } } Ok(p) } @@ -284,5 +290,11 @@ mod tests { // group of the calling process. process.pid = 0; assert!(process.signal(libc::SIGCONT).is_ok()); + + if cfg!(feature = "standard-oci-runtime") { + assert_eq!(process.stdin.unwrap(), std::io::stdin().as_raw_fd()); + assert_eq!(process.stdout.unwrap(), std::io::stdout().as_raw_fd()); + assert_eq!(process.stderr.unwrap(), std::io::stderr().as_raw_fd()); + } } } diff --git a/src/agent/rustjail/src/utils.rs b/src/agent/rustjail/src/utils.rs deleted file mode 100644 index cf69c77a2..000000000 --- a/src/agent/rustjail/src/utils.rs +++ /dev/null @@ -1,120 +0,0 @@ -// Copyright (c) 2021 Ant Group -// -// SPDX-License-Identifier: Apache-2.0 -// -use anyhow::{anyhow, Context, Result}; -use libc::gid_t; -use libc::uid_t; -use std::fs::File; -use std::io::{BufRead, BufReader}; - -const PASSWD_FILE: &str = "/etc/passwd"; - -// An entry from /etc/passwd -#[derive(Debug, PartialEq, PartialOrd)] -pub struct PasswdEntry { - // username - pub name: String, - // user password - pub passwd: String, - // user id - pub uid: uid_t, - // group id - pub gid: gid_t, - // user Information - pub gecos: String, - // home directory - pub dir: String, - // User's Shell - pub shell: String, -} - -// get an entry for a given `uid` from `/etc/passwd` -fn get_entry_by_uid(uid: uid_t, path: &str) -> Result { - let file = File::open(path).with_context(|| format!("open file {}", path))?; - let mut reader = BufReader::new(file); - - let mut line = String::new(); - loop { - line.clear(); - match reader.read_line(&mut line) { - Ok(0) => return Err(anyhow!(format!("file {} is empty", path))), - Ok(_) => (), - Err(e) => { - return Err(anyhow!(format!( - "failed to read file {} with {:?}", - path, e - ))) - } - } - - if line.starts_with('#') { - continue; - } - - let parts: Vec<&str> = line.split(':').map(|part| part.trim()).collect(); - if parts.len() != 7 { - continue; - } - - match parts[2].parse() { - Err(_e) => continue, - Ok(new_uid) => { - if uid != new_uid { - continue; - } - - let entry = PasswdEntry { - name: parts[0].to_string(), - passwd: parts[1].to_string(), - uid: new_uid, - gid: parts[3].parse().unwrap_or(0), - gecos: parts[4].to_string(), - dir: parts[5].to_string(), - shell: parts[6].to_string(), - }; - - return Ok(entry); - } - } - } -} - -pub fn home_dir(uid: uid_t) -> Result { - get_entry_by_uid(uid, PASSWD_FILE).map(|entry| entry.dir) -} - -#[cfg(test)] -mod tests { - use super::*; - use std::io::Write; - use tempfile::Builder; - - #[test] - fn test_get_entry_by_uid() { - let tmpdir = Builder::new().tempdir().unwrap(); - let tmpdir_path = tmpdir.path().to_str().unwrap(); - let temp_passwd = format!("{}/passwd", tmpdir_path); - - let mut tempf = File::create(temp_passwd.as_str()).unwrap(); - let passwd_entries = "root:x:0:0:root:/root0:/bin/bash -root:x:1:0:root:/root1:/bin/bash -#root:x:1:0:root:/rootx:/bin/bash -root:x:2:0:root:/root2:/bin/bash -root:x:3:0:root:/root3 -root:x:3:0:root:/root3:/bin/bash"; - writeln!(tempf, "{}", passwd_entries).unwrap(); - - let entry = get_entry_by_uid(0, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root0"); - - let entry = get_entry_by_uid(1, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root1"); - - let entry = get_entry_by_uid(2, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root2"); - - let entry = get_entry_by_uid(3, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root3"); - } -} diff --git a/src/agent/src/mount.rs b/src/agent/src/mount.rs index d4729c881..ff24f4957 100644 --- a/src/agent/src/mount.rs +++ b/src/agent/src/mount.rs @@ -859,8 +859,9 @@ pub fn get_mount_fs_type_from_file(mount_file: &str, mount_point: &str) -> Resul } Err(anyhow!( - "failed to find FS type for mount point {}", - mount_point + "failed to find FS type for mount point {}, mount file content: {:?}", + mount_point, + fs::read_to_string(mount_file) )) } diff --git a/src/agent/src/rpc.rs b/src/agent/src/rpc.rs index f7775620d..070d3c0db 100644 --- a/src/agent/src/rpc.rs +++ b/src/agent/src/rpc.rs @@ -23,8 +23,9 @@ use cgroups::freezer::FreezerState; use oci::{LinuxNamespace, Root, Spec}; use protobuf::{Message, RepeatedField, SingularPtrField}; use protocols::agent::{ - AddSwapRequest, AgentDetails, CopyFileRequest, GuestDetailsResponse, Interfaces, Metrics, - OOMEvent, ReadStreamResponse, Routes, StatsContainerResponse, VolumeStatsRequest, + AddSwapRequest, AgentDetails, CopyFileRequest, GetIPTablesRequest, GetIPTablesResponse, + GuestDetailsResponse, Interfaces, Metrics, OOMEvent, ReadStreamResponse, Routes, + SetIPTablesRequest, SetIPTablesResponse, StatsContainerResponse, VolumeStatsRequest, WaitProcessResponse, WriteStreamResponse, }; use protocols::csi::{VolumeCondition, VolumeStatsResponse, VolumeUsage, VolumeUsage_Unit}; @@ -76,7 +77,7 @@ use std::time::Duration; use nix::unistd::{Gid, Uid}; use std::fs::{File, OpenOptions}; -use std::io::{BufRead, BufReader}; +use std::io::{BufRead, BufReader, Write}; use std::os::unix::fs::FileExt; use std::path::PathBuf; @@ -85,11 +86,22 @@ const MODPROBE_PATH: &str = "/sbin/modprobe"; const ANNO_K8S_IMAGE_NAME: &str = "io.kubernetes.cri.image-name"; const CONFIG_JSON: &str = "config.json"; +const IPTABLES_SAVE: &str = "/sbin/iptables-save"; +const IPTABLES_RESTORE: &str = "/sbin/iptables-restore"; +const IP6TABLES_SAVE: &str = "/sbin/ip6tables-save"; +const IP6TABLES_RESTORE: &str = "/sbin/ip6tables-restore"; + const ERR_CANNOT_GET_WRITER: &str = "Cannot get writer"; const ERR_INVALID_BLOCK_SIZE: &str = "Invalid block size"; const ERR_NO_LINUX_FIELD: &str = "Spec does not contain linux field"; const ERR_NO_SANDBOX_PIDNS: &str = "Sandbox does not have sandbox_pidns"; +// IPTABLES_RESTORE_WAIT_SEC is the timeout value provided to iptables-restore --wait. Since we +// don't expect other writers to iptables, we don't expect contention for grabbing the iptables +// filesystem lock. Based on this, 5 seconds seems a resonable timeout period in case the lock is +// not available. +const IPTABLES_RESTORE_WAIT_SEC: u64 = 5; + // Convenience macro to obtain the scope logger macro_rules! sl { () => { @@ -1061,6 +1073,140 @@ impl protocols::agent_ttrpc::AgentService for AgentService { }) } + async fn get_ip_tables( + &self, + ctx: &TtrpcContext, + req: GetIPTablesRequest, + ) -> ttrpc::Result { + trace_rpc_call!(ctx, "get_iptables", req); + is_allowed!(req); + + info!(sl!(), "get_ip_tables: request received"); + + let cmd = if req.is_ipv6 { + IP6TABLES_SAVE + } else { + IPTABLES_SAVE + } + .to_string(); + + match Command::new(cmd.clone()).output() { + Ok(output) => Ok(GetIPTablesResponse { + data: output.stdout, + ..Default::default() + }), + Err(e) => { + warn!(sl!(), "failed to run {}: {:?}", cmd, e.kind()); + return Err(ttrpc_error!(ttrpc::Code::INTERNAL, e)); + } + } + } + + async fn set_ip_tables( + &self, + ctx: &TtrpcContext, + req: SetIPTablesRequest, + ) -> ttrpc::Result { + trace_rpc_call!(ctx, "set_iptables", req); + is_allowed!(req); + + info!(sl!(), "set_ip_tables request received"); + + let cmd = if req.is_ipv6 { + IP6TABLES_RESTORE + } else { + IPTABLES_RESTORE + } + .to_string(); + + let mut child = match Command::new(cmd.clone()) + .arg("--wait") + .arg(IPTABLES_RESTORE_WAIT_SEC.to_string()) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + { + Ok(child) => child, + Err(e) => { + warn!(sl!(), "failure to spawn {}: {:?}", cmd, e.kind()); + return Err(ttrpc_error!(ttrpc::Code::INTERNAL, e)); + } + }; + + let mut stdin = match child.stdin.take() { + Some(si) => si, + None => { + println!("failed to get stdin from child"); + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + "failed to take stdin from child".to_string() + )); + } + }; + + let (tx, rx) = tokio::sync::oneshot::channel::(); + let handle = tokio::spawn(async move { + let _ = match stdin.write_all(&req.data) { + Ok(o) => o, + Err(e) => { + warn!(sl!(), "error writing stdin: {:?}", e.kind()); + return; + } + }; + if tx.send(1).is_err() { + warn!(sl!(), "stdin writer thread receiver dropped"); + }; + }); + + if tokio::time::timeout(Duration::from_secs(IPTABLES_RESTORE_WAIT_SEC), rx) + .await + .is_err() + { + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + "timeout waiting for stdin writer to complete".to_string() + )); + } + + if handle.await.is_err() { + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + "stdin writer thread failure".to_string() + )); + } + + let output = match child.wait_with_output() { + Ok(o) => o, + Err(e) => { + warn!( + sl!(), + "failure waiting for spawned {} to complete: {:?}", + cmd, + e.kind() + ); + return Err(ttrpc_error!(ttrpc::Code::INTERNAL, e)); + } + }; + + if !output.status.success() { + warn!(sl!(), "{} failed: {:?}", cmd, output.stderr); + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + format!( + "{} failed: {:?}", + cmd, + String::from_utf8_lossy(&output.stderr) + ) + )); + } + + Ok(SetIPTablesResponse { + data: output.stdout, + ..Default::default() + }) + } + async fn list_interfaces( &self, ctx: &TtrpcContext, @@ -1960,6 +2106,7 @@ mod tests { skip_if_not_root, }; use nix::mount; + use nix::sched::{unshare, CloneFlags}; use oci::{Hook, Hooks, Linux, LinuxNamespace}; use tempfile::{tempdir, TempDir}; use ttrpc::{r#async::TtrpcContext, MessageHeader}; @@ -2896,4 +3043,162 @@ OtherField:other assert_eq!(stats.used, 3); assert_eq!(stats.available, available - 2); } + + #[tokio::test] + async fn test_ip_tables() { + skip_if_not_root!(); + + let logger = slog::Logger::root(slog::Discard, o!()); + let sandbox = Sandbox::new(&logger).unwrap(); + let agent_service = Box::new(AgentService { + sandbox: Arc::new(Mutex::new(sandbox)), + }); + + let ctx = mk_ttrpc_context(); + + // Move to a new netns in order to ensure we don't trash the hosts' iptables + unshare(CloneFlags::CLONE_NEWNET).unwrap(); + + // Get initial iptables, we expect to be empty: + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: false, + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "get ip tables should succeed"); + assert_eq!( + result.unwrap().data.len(), + 0, + "ip tables should be empty initially" + ); + + // Initial ip6 ip tables should also be empty: + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: true, + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "get ip6 tables should succeed"); + assert_eq!( + result.unwrap().data.len(), + 0, + "ip tables should be empty initially" + ); + + // Verify that attempting to write 'empty' iptables results in no error: + let empty_rules = ""; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: false, + data: empty_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "set ip tables with no data should succeed"); + + // Verify that attempting to write "garbage" iptables results in an error: + let garbage_rules = r#" +this +is +just garbage +"#; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: false, + data: garbage_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_err(), "set iptables with garbage should fail"); + + // Verify setup of valid iptables:Setup valid set of iptables: + let valid_rules = r#" +*nat +-A PREROUTING -d 192.168.103.153/32 -j DNAT --to-destination 192.168.188.153 + +COMMIT + +"#; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: false, + data: valid_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "set ip tables should succeed"); + + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: false, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!(!result.data.is_empty(), "we should have non-zero output:"); + assert!( + std::str::from_utf8(&*result.data).unwrap().contains( + "PREROUTING -d 192.168.103.153/32 -j DNAT --to-destination 192.168.188.153" + ), + "We should see the resulting rule" + ); + + // Verify setup of valid ip6tables: + let valid_ipv6_rules = r#" +*filter +-A INPUT -s 2001:db8:100::1/128 -i sit+ -p tcp -m tcp --sport 512:65535 + +COMMIT + +"#; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: true, + data: valid_ipv6_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "set ip6 tables should succeed"); + + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: true, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!(!result.data.is_empty(), "we should have non-zero output:"); + assert!( + std::str::from_utf8(&*result.data) + .unwrap() + .contains("INPUT -s 2001:db8:100::1/128 -i sit+ -p tcp -m tcp --sport 512:65535"), + "We should see the resulting rule" + ); + } } diff --git a/src/libs/Cargo.lock b/src/libs/Cargo.lock index ad9d8672c..fc7fc5da5 100644 --- a/src/libs/Cargo.lock +++ b/src/libs/Cargo.lock @@ -1,11 +1,30 @@ # This file is automatically @generated by Cargo. # It is not intended for manual editing. +version = 3 + +[[package]] +name = "anyhow" +version = "1.0.57" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08f9b8508dccb7687a1d6c4ce66b2b0ecef467c94667de27d8d7fe1f8d2a9cdc" + [[package]] name = "arc-swap" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c5d78ce20460b82d3fa150275ed9d55e21064fc7951177baacf86a145c4a4b1f" +[[package]] +name = "async-trait" +version = "0.1.53" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed6aa3524a2dfcf9fe180c51eae2b58738348d819517ceadf95789c51fff7600" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "autocfg" version = "1.0.1" @@ -14,9 +33,37 @@ checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a" [[package]] name = "bitflags" -version = "1.3.2" +version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" +checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693" + +[[package]] +name = "byteorder" +version = "1.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610" + +[[package]] +name = "bytes" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "206fdffcfa2df7cbe15601ef46c813fce0965eb3286db6b56c583b814b51c81c" +dependencies = [ + "byteorder", + "iovec", +] + +[[package]] +name = "bytes" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4872d67bab6358e59559027aa3b9157c53d9358c51423c17554809a8858e0f8" + +[[package]] +name = "cc" +version = "1.0.73" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2fff2a6927b3bb87f9595d67196a70493f627687a71d87a0d692242c33f58c11" [[package]] name = "cfg-if" @@ -57,6 +104,23 @@ dependencies = [ "lazy_static", ] +[[package]] +name = "derive-new" +version = "0.5.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3418329ca0ad70234b9735dc4ceed10af4df60eff9c8e7b06cb5e520d92c3535" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "either" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e78d4f1cc4ae33bbfc157ed5d5a5ef3bc29227303d595861deb238fcec4e9457" + [[package]] name = "fastrand" version = "1.6.0" @@ -66,6 +130,126 @@ dependencies = [ "instant", ] +[[package]] +name = "fixedbitset" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37ab347416e802de484e4d03c7316c48f1ecb56574dfd4a46a80f173ce1de04d" + +[[package]] +name = "futures" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f73fe65f54d1e12b726f517d3e2135ca3125a437b6d998caf1962961f7172d9e" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3083ce4b914124575708913bca19bfe887522d6e2e6d0952943f5eac4a74010" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c09fd04b7e4073ac7156a9539b57a484a8ea920f79c7c675d05d289ab6110d3" + +[[package]] +name = "futures-executor" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9420b90cfa29e327d0429f19be13e7ddb68fa1cccb09d65e5706b8c7a749b8a6" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc4045962a5a5e935ee2fdedaa4e08284547402885ab326734432bed5d12966b" + +[[package]] +name = "futures-macro" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33c1e13800337f4d4d7a316bf45a567dbcb6ffe087f16424852d97e97a91f512" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "futures-sink" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "21163e139fa306126e6eedaf49ecdb4588f939600f0b1e770f4205ee4b7fa868" + +[[package]] +name = "futures-task" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c66a976bf5909d801bbef33416c41372779507e7a6b3a5e25e4749c58f776a" + +[[package]] +name = "futures-util" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d8b7abd5d659d9b90c8cba917f6ec750a74e2dc23902ef9cd4cc8c8b22e6036a" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "pin-utils", + "slab", +] + +[[package]] +name = "hashbrown" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab5ef0d4909ef3724cc8cce6ccc8572c5c817592e9285f5464f8e86f8bd3726e" + +[[package]] +name = "heck" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d621efb26863f0e9924c6ac577e8275e5e6b77455db64ffa6c65c904e9e132c" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "indexmap" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f647032dfaa1f8b6dc29bd3edb7bbef4861b8b8007ebb118d6db284fd59f6ee" +dependencies = [ + "autocfg", + "hashbrown", +] + [[package]] name = "instant" version = "0.1.12" @@ -75,6 +259,24 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "iovec" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2b3ea6ff95e175473f8ffe6a7eb7c00d054240321b84c57051175fe3c1e075e" +dependencies = [ + "libc", +] + +[[package]] +name = "itertools" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9a9d19fa1e79b6215ff29b9d6880b706147f16e9b1dbb1e4e5947b5b02bc5e3" +dependencies = [ + "either", +] + [[package]] name = "itoa" version = "1.0.1" @@ -89,9 +291,18 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" [[package]] name = "libc" -version = "0.2.113" +version = "0.2.124" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eef78b64d87775463c549fbd80e19249ef436ea3bf1de2a1eb7e717ec7fab1e9" +checksum = "21a41fed9d98f27ab1c6d161da622a4fa35e8a54a8adc24bbf3ddd0ef70b0e50" + +[[package]] +name = "log" +version = "0.4.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6389c490849ff5bc16be905ae24bc913a9c8892e19b2341dbc175e14c341c2b8" +dependencies = [ + "cfg-if", +] [[package]] name = "logging" @@ -105,6 +316,85 @@ dependencies = [ "tempfile", ] +[[package]] +name = "memchr" +version = "2.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "308cc39be01b73d0d18f82a0e7b2a3df85245f84af96fdddc5d202d27e47b86a" + +[[package]] +name = "memoffset" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5aa361d4faea93603064a027415f07bd8e1d5c88c9fbf68bf56a285428fd79ce" +dependencies = [ + "autocfg", +] + +[[package]] +name = "mio" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52da4364ffb0e4fe33a9841a98a3f3014fb964045ce4f7a45a398243c8d6b0c9" +dependencies = [ + "libc", + "log", + "miow", + "ntapi", + "wasi 0.11.0+wasi-snapshot-preview1", + "winapi", +] + +[[package]] +name = "miow" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9f1c5b025cda876f66ef43a113f91ebc9f4ccef34843000e0adf6ebbab84e21" +dependencies = [ + "winapi", +] + +[[package]] +name = "multimap" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5ce46fe64a9d73be07dcbe690a38ce1b293be448fd8ce1e6c1b8062c9f72c6a" + +[[package]] +name = "nix" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f5e06129fb611568ef4e868c14b326274959aa70ff7776e9d55323531c374945" +dependencies = [ + "bitflags", + "cc", + "cfg-if", + "libc", + "memoffset", +] + +[[package]] +name = "nix" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f866317acbd3a240710c63f065ffb1e4fd466259045ccb504130b7f668f35c6" +dependencies = [ + "bitflags", + "cc", + "cfg-if", + "libc", + "memoffset", +] + +[[package]] +name = "ntapi" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c28774a7fd2fbb4f0babd8237ce554b73af68021b5f695a3cebd6c59bac0980f" +dependencies = [ + "winapi", +] + [[package]] name = "num-integer" version = "0.1.44" @@ -130,6 +420,138 @@ version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "da32515d9f6e6e489d7bc9d84c71b060db7247dc035bbe44eac88cf87486d8d5" +[[package]] +name = "petgraph" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "467d164a6de56270bd7c4d070df81d07beace25012d5103ced4e9ff08d6afdb7" +dependencies = [ + "fixedbitset", + "indexmap", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e280fbe77cc62c91527259e9442153f4688736748d24660126286329742b4c6c" + +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + +[[package]] +name = "proc-macro2" +version = "1.0.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec757218438d5fda206afc041538b2f6d889286160d649a86a24d37e1235afd1" +dependencies = [ + "unicode-xid", +] + +[[package]] +name = "prost" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de5e2533f59d08fcf364fd374ebda0692a70bd6d7e66ef97f306f45c6c5d8020" +dependencies = [ + "bytes 1.1.0", + "prost-derive", +] + +[[package]] +name = "prost-build" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "355f634b43cdd80724ee7848f95770e7e70eefa6dcf14fea676216573b8fd603" +dependencies = [ + "bytes 1.1.0", + "heck", + "itertools", + "log", + "multimap", + "petgraph", + "prost", + "prost-types", + "tempfile", + "which", +] + +[[package]] +name = "prost-derive" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "600d2f334aa05acb02a755e217ef1ab6dea4d51b58b7846588b747edec04efba" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "prost-types" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "603bbd6394701d13f3f25aada59c7de9d35a6a5887cfc156181234a44002771b" +dependencies = [ + "bytes 1.1.0", + "prost", +] + +[[package]] +name = "protobuf" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e86d370532557ae7573551a1ec8235a0f8d6cb276c7c9e6aa490b511c447485" +dependencies = [ + "serde", + "serde_derive", +] + +[[package]] +name = "protobuf-codegen" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de113bba758ccf2c1ef816b127c958001b7831136c9bc3f8e9ec695ac4e82b0c" +dependencies = [ + "protobuf", +] + +[[package]] +name = "protobuf-codegen-pure" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d1a4febc73bf0cada1d77c459a0c8e5973179f1cfd5b0f1ab789d45b17b6440" +dependencies = [ + "protobuf", + "protobuf-codegen", +] + +[[package]] +name = "protocols" +version = "0.1.0" +dependencies = [ + "async-trait", + "protobuf", + "serde", + "serde_json", + "ttrpc", + "ttrpc-codegen", +] + +[[package]] +name = "quote" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1feb54ed693b93a84e14094943b84b7c4eae204c512b7ccb95ab0c66d278ad1" +dependencies = [ + "proc-macro2", +] + [[package]] name = "redox_syscall" version = "0.2.10" @@ -167,6 +589,20 @@ name = "serde" version = "1.0.133" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97565067517b60e2d1ea8b268e59ce036de907ac523ad83a0475da04e818989a" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.133" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed201699328568d8d08208fdd080e3ff594e6c422e438b6705905da01005d537" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] [[package]] name = "serde_json" @@ -179,6 +615,12 @@ dependencies = [ "serde", ] +[[package]] +name = "slab" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb703cfe953bccee95685111adeedb76fabe4e97549a58d16f03ea7b9367bb32" + [[package]] name = "slog" version = "2.7.0" @@ -220,6 +662,27 @@ dependencies = [ "slog", ] +[[package]] +name = "socket2" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66d72b759436ae32898a2af0a14218dbf55efde3feeb170eb623637db85ee1e0" +dependencies = [ + "libc", + "winapi", +] + +[[package]] +name = "syn" +version = "1.0.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b683b2b825c8eef438b77c36a06dc262294da3d5a5813fac20da149241dcd44d" +dependencies = [ + "proc-macro2", + "quote", + "unicode-xid", +] + [[package]] name = "take_mut" version = "0.2.2" @@ -240,6 +703,26 @@ dependencies = [ "winapi", ] +[[package]] +name = "thiserror" +version = "1.0.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "854babe52e4df1653706b98fcfc05843010039b406875930a70e4d9644e5c417" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aa32fd3f627f367fe16f893e2597ae3c05020f8bba2666a4e6ea73d377e5714b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "thread_local" version = "1.1.3" @@ -256,16 +739,141 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6db9e6914ab8b1ae1c260a4ae7a49b6c5611b40328a735b21862567685e73255" dependencies = [ "libc", - "wasi", + "wasi 0.10.0+wasi-snapshot-preview1", "winapi", ] +[[package]] +name = "tokio" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af73ac49756f3f7c01172e34a23e5d0216f6c32333757c2c61feb2bbff5a5ee" +dependencies = [ + "bytes 1.1.0", + "libc", + "memchr", + "mio", + "pin-project-lite", + "socket2", + "tokio-macros", + "winapi", +] + +[[package]] +name = "tokio-macros" +version = "1.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b557f72f448c511a979e2564e55d74e6c4432fc96ff4f6241bc6bded342643b7" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tokio-vsock" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e0723fc001950a3b018947b05eeb45014fd2b7c6e8f292502193ab74486bdb6" +dependencies = [ + "bytes 0.4.12", + "futures", + "libc", + "tokio", + "vsock", +] + +[[package]] +name = "ttrpc" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66a973ce6d5eaa20c173635b29ffb660dafbc7ef109172c0015ba44e47a23711" +dependencies = [ + "async-trait", + "byteorder", + "futures", + "libc", + "log", + "nix 0.20.2", + "protobuf", + "protobuf-codegen-pure", + "thiserror", + "tokio", + "tokio-vsock", +] + +[[package]] +name = "ttrpc-codegen" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "809eda4e459820237104e4b61d6b41bbe6c9e1ce6adf4057955e6e6722a90408" +dependencies = [ + "protobuf", + "protobuf-codegen", + "protobuf-codegen-pure", + "ttrpc-compiler", +] + +[[package]] +name = "ttrpc-compiler" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2978ed3fa047d8fd55cbeb4d4a61d461fb3021a90c9618519c73ce7e5bb66c15" +dependencies = [ + "derive-new", + "prost", + "prost-build", + "prost-types", + "protobuf", + "protobuf-codegen", + "tempfile", +] + +[[package]] +name = "unicode-segmentation" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e8820f5d777f6224dc4be3632222971ac30164d4a258d595640799554ebfd99" + +[[package]] +name = "unicode-xid" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3" + +[[package]] +name = "vsock" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e32675ee2b3ce5df274c0ab52d19b28789632406277ca26bffee79a8e27dc133" +dependencies = [ + "libc", + "nix 0.23.1", +] + [[package]] name = "wasi" version = "0.10.0+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1a143597ca7c7793eff794def352d41792a93c481eb1042423ff7ff72ba2c31f" +[[package]] +name = "wasi" +version = "0.11.0+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" + +[[package]] +name = "which" +version = "4.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c4fb54e6113b6a8772ee41c3404fb0301ac79604489467e0a9ce1f3e97c24ae" +dependencies = [ + "either", + "lazy_static", + "libc", +] + [[package]] name = "winapi" version = "0.3.9" diff --git a/src/libs/Cargo.toml b/src/libs/Cargo.toml index f4b707a14..16eedb91f 100644 --- a/src/libs/Cargo.toml +++ b/src/libs/Cargo.toml @@ -2,5 +2,6 @@ members = [ "logging", "safe-path", + "protocols", ] resolver = "2" diff --git a/src/libs/protocols/protos/agent.proto b/src/libs/protocols/protos/agent.proto index 13a9094a8..504cae83a 100644 --- a/src/libs/protocols/protos/agent.proto +++ b/src/libs/protocols/protos/agent.proto @@ -51,6 +51,8 @@ service AgentService { rpc ListInterfaces(ListInterfacesRequest) returns(Interfaces); rpc ListRoutes(ListRoutesRequest) returns (Routes); rpc AddARPNeighbors(AddARPNeighborsRequest) returns (google.protobuf.Empty); + rpc GetIPTables(GetIPTablesRequest) returns (GetIPTablesResponse); + rpc SetIPTables(SetIPTablesRequest) returns (SetIPTablesResponse); // observability rpc GetMetrics(GetMetricsRequest) returns (Metrics); @@ -328,6 +330,28 @@ message AddARPNeighborsRequest { ARPNeighbors neighbors = 1; } +message GetIPTablesRequest { + bool is_ipv6 = 1; +} + +message GetIPTablesResponse{ + // raw stdout from iptables-save or ip6tables-save + bytes data = 1; +} + +message SetIPTablesRequest { + bool is_ipv6 = 1; + + // iptables, in raw format expected to be passed to stdin + // of iptables-save or ip6tables-save + bytes data = 2; +} + +message SetIPTablesResponse{ + // raw stdout from iptables-restore or ip6tables-restore + bytes data = 1; +} + message OnlineCPUMemRequest { // Wait specifies if the caller waits for the agent to online all resources. // If true the agent returns once all resources have been connected, otherwise all diff --git a/src/runtime/cmd/kata-runtime/kata-iptables.go b/src/runtime/cmd/kata-runtime/kata-iptables.go new file mode 100644 index 000000000..8e2bd93c6 --- /dev/null +++ b/src/runtime/cmd/kata-runtime/kata-iptables.go @@ -0,0 +1,122 @@ +// Copyright (c) 2022 Apple Inc. +// +// SPDX-License-Identifier: Apache-2.0 +// + +package main + +import ( + "fmt" + "io/ioutil" + + containerdshim "github.com/kata-containers/kata-containers/src/runtime/pkg/containerd-shim-v2" + "github.com/kata-containers/kata-containers/src/runtime/pkg/katautils" + "github.com/kata-containers/kata-containers/src/runtime/pkg/utils/shimclient" + "github.com/urfave/cli" +) + +var ( + sandboxID string + isIPv6 bool +) +var iptablesSubCmds = []cli.Command{ + getIPTablesCommand, + setIPTablesCommand, +} + +var kataIPTablesCommand = cli.Command{ + Name: "iptables", + Usage: "get or set iptables within the Kata Containers guest", + Subcommands: iptablesSubCmds, + Action: func(context *cli.Context) { + cli.ShowSubcommandHelp(context) + }, +} + +var getIPTablesCommand = cli.Command{ + Name: "get", + Usage: "get iptables from the Kata Containers guest", + Flags: []cli.Flag{ + cli.StringFlag{ + Name: "sandbox-id", + Usage: "the target sandbox for getting the iptables", + Required: true, + Destination: &sandboxID, + }, + cli.BoolFlag{ + Name: "v6", + Usage: "indicate we're requesting ipv6 iptables", + Destination: &isIPv6, + }, + }, + Action: func(c *cli.Context) error { + // verify sandbox exists: + if err := katautils.VerifyContainerID(sandboxID); err != nil { + return err + } + + url := containerdshim.IPTablesUrl + if isIPv6 { + url = containerdshim.IP6TablesUrl + } + body, err := shimclient.DoGet(sandboxID, defaultTimeout, url) + if err != nil { + return err + } + + fmt.Println(string(body)) + return nil + }, +} + +var setIPTablesCommand = cli.Command{ + Name: "set", + Usage: "set iptables in a specifc Kata Containers guest based on file", + Flags: []cli.Flag{ + cli.StringFlag{ + Name: "sandbox-id", + Usage: "the target sandbox for setting the iptables", + Required: true, + Destination: &sandboxID, + }, + cli.BoolFlag{ + Name: "v6", + Usage: "indicate we're requesting ipv6 iptables", + Destination: &isIPv6, + }, + }, + Action: func(c *cli.Context) error { + iptablesFile := c.Args().Get(0) + + // verify sandbox exists: + if err := katautils.VerifyContainerID(sandboxID); err != nil { + return err + } + + // verify iptables were provided: + if iptablesFile == "" { + return fmt.Errorf("iptables file not provided") + } + + if !katautils.FileExists(iptablesFile) { + return fmt.Errorf("iptables file does not exist: %s", iptablesFile) + } + + // Read file into buffer, and make request to the appropriate shim + buf, err := ioutil.ReadFile(iptablesFile) + if err != nil { + return err + } + + url := containerdshim.IPTablesUrl + if isIPv6 { + url = containerdshim.IP6TablesUrl + } + + if err = shimclient.DoPut(sandboxID, defaultTimeout, url, "application/octet-stream", buf); err != nil { + return fmt.Errorf("Error observed when making iptables-set request(%s): %s", iptablesFile, err) + } + + return nil + }, +} diff --git a/src/runtime/cmd/kata-runtime/kata-volume.go b/src/runtime/cmd/kata-runtime/kata-volume.go index 55274f7d8..b33febe19 100644 --- a/src/runtime/cmd/kata-runtime/kata-volume.go +++ b/src/runtime/cmd/kata-runtime/kata-volume.go @@ -163,5 +163,5 @@ func Resize(volumePath string, size uint64) error { if err != nil { return err } - return shimclient.DoPost(sandboxId, defaultTimeout, containerdshim.DirectVolumeResizeUrl, encoded) + return shimclient.DoPost(sandboxId, defaultTimeout, containerdshim.DirectVolumeResizeUrl, "application/json", encoded) } diff --git a/src/runtime/cmd/kata-runtime/main.go b/src/runtime/cmd/kata-runtime/main.go index def7431f0..a722afd6b 100644 --- a/src/runtime/cmd/kata-runtime/main.go +++ b/src/runtime/cmd/kata-runtime/main.go @@ -125,6 +125,7 @@ var runtimeCommands = []cli.Command{ kataMetricsCLICommand, factoryCLICommand, kataVolumeCommand, + kataIPTablesCommand, } // runtimeBeforeSubcommands is the function to run before command-line diff --git a/src/runtime/pkg/containerd-shim-v2/shim_management.go b/src/runtime/pkg/containerd-shim-v2/shim_management.go index e10922250..a6b9bdba3 100644 --- a/src/runtime/pkg/containerd-shim-v2/shim_management.go +++ b/src/runtime/pkg/containerd-shim-v2/shim_management.go @@ -29,13 +29,17 @@ import ( "github.com/prometheus/client_golang/prometheus" dto "github.com/prometheus/client_model/go" "github.com/prometheus/common/expfmt" + "github.com/sirupsen/logrus" ) const ( - DirectVolumePathKey = "path" - + DirectVolumePathKey = "path" + AgentUrl = "/agent-url" DirectVolumeStatUrl = "/direct-volume/stats" DirectVolumeResizeUrl = "/direct-volume/resize" + IPTablesUrl = "/iptables" + IP6TablesUrl = "/ip6tables" + MetricsUrl = "/metrics" ) var ( @@ -195,6 +199,48 @@ func (s *service) serveVolumeResize(w http.ResponseWriter, r *http.Request) { w.Write([]byte("")) } +func (s *service) ip6TablesHandler(w http.ResponseWriter, r *http.Request) { + s.genericIPTablesHandler(w, r, true) +} + +func (s *service) ipTablesHandler(w http.ResponseWriter, r *http.Request) { + s.genericIPTablesHandler(w, r, false) +} + +func (s *service) genericIPTablesHandler(w http.ResponseWriter, r *http.Request, isIPv6 bool) { + logger := shimMgtLog.WithFields(logrus.Fields{"handler": "iptables", "ipv6": isIPv6}) + + switch r.Method { + case http.MethodPut: + body, err := ioutil.ReadAll(r.Body) + if err != nil { + logger.WithError(err).Error("failed to read request body") + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(err.Error())) + return + } + + if err = s.sandbox.SetIPTables(context.Background(), isIPv6, body); err != nil { + logger.WithError(err).Error("failed to set IPTables") + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(err.Error())) + } + w.Write([]byte("")) + + case http.MethodGet: + buf, err := s.sandbox.GetIPTables(context.Background(), isIPv6) + if err != nil { + logger.WithError(err).Error("failed to get IPTables") + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(err.Error())) + } + w.Write(buf) + default: + w.WriteHeader(http.StatusNotImplemented) + return + } +} + func (s *service) startManagementServer(ctx context.Context, ociSpec *specs.Spec) { // metrics socket will under sandbox's bundle path metricsAddress := SocketAddress(s.id) @@ -215,10 +261,12 @@ func (s *service) startManagementServer(ctx context.Context, ociSpec *specs.Spec // bind handler m := http.NewServeMux() - m.Handle("/metrics", http.HandlerFunc(s.serveMetrics)) - m.Handle("/agent-url", http.HandlerFunc(s.agentURL)) + m.Handle(MetricsUrl, http.HandlerFunc(s.serveMetrics)) + m.Handle(AgentUrl, http.HandlerFunc(s.agentURL)) m.Handle(DirectVolumeStatUrl, http.HandlerFunc(s.serveVolumeStats)) m.Handle(DirectVolumeResizeUrl, http.HandlerFunc(s.serveVolumeResize)) + m.Handle(IPTablesUrl, http.HandlerFunc(s.ipTablesHandler)) + m.Handle(IP6TablesUrl, http.HandlerFunc(s.ip6TablesHandler)) s.mountPprofHandle(m, ociSpec) // register shim metrics diff --git a/src/runtime/pkg/containerd-shim-v2/wait.go b/src/runtime/pkg/containerd-shim-v2/wait.go index 0f6c4fe06..5e86149c1 100644 --- a/src/runtime/pkg/containerd-shim-v2/wait.go +++ b/src/runtime/pkg/containerd-shim-v2/wait.go @@ -78,7 +78,7 @@ func wait(ctx context.Context, s *service, c *container, execID string) (int32, shimLog.WithField("sandbox", s.sandbox.ID()).Error("failed to delete sandbox") } } else { - if _, err = s.sandbox.StopContainer(ctx, c.id, false); err != nil { + if _, err = s.sandbox.StopContainer(ctx, c.id, true); err != nil { shimLog.WithError(err).WithField("container", c.id).Warn("stop container failed") } } diff --git a/src/runtime/pkg/kata-monitor/metrics.go b/src/runtime/pkg/kata-monitor/metrics.go index e5d947767..98ecb68f0 100644 --- a/src/runtime/pkg/kata-monitor/metrics.go +++ b/src/runtime/pkg/kata-monitor/metrics.go @@ -15,6 +15,7 @@ import ( "sync" "time" + containerdshim "github.com/kata-containers/kata-containers/src/runtime/pkg/containerd-shim-v2" mutils "github.com/kata-containers/kata-containers/src/runtime/pkg/utils" "github.com/kata-containers/kata-containers/src/runtime/pkg/utils/shimclient" "github.com/prometheus/client_golang/prometheus" @@ -239,7 +240,7 @@ func (km *KataMonitor) aggregateSandboxMetrics(encoder expfmt.Encoder) error { } func getParsedMetrics(sandboxID string, sandboxMetadata sandboxCRIMetadata) ([]*dto.MetricFamily, error) { - body, err := shimclient.DoGet(sandboxID, defaultTimeout, "metrics") + body, err := shimclient.DoGet(sandboxID, defaultTimeout, containerdshim.MetricsUrl) if err != nil { return nil, err } @@ -249,7 +250,7 @@ func getParsedMetrics(sandboxID string, sandboxMetadata sandboxCRIMetadata) ([]* // GetSandboxMetrics will get sandbox's metrics from shim func GetSandboxMetrics(sandboxID string) (string, error) { - body, err := shimclient.DoGet(sandboxID, defaultTimeout, "metrics") + body, err := shimclient.DoGet(sandboxID, defaultTimeout, containerdshim.MetricsUrl) if err != nil { return "", err } diff --git a/src/runtime/pkg/utils/shimclient/shim_management_client.go b/src/runtime/pkg/utils/shimclient/shim_management_client.go index c9ed3ad69..3f9e68650 100644 --- a/src/runtime/pkg/utils/shimclient/shim_management_client.go +++ b/src/runtime/pkg/utils/shimclient/shim_management_client.go @@ -9,6 +9,7 @@ import ( "bytes" "fmt" "io" + "io/ioutil" "net" "net/http" "time" @@ -48,7 +49,7 @@ func DoGet(sandboxID string, timeoutInSeconds time.Duration, urlPath string) ([] return nil, err } - resp, err := client.Get(fmt.Sprintf("http://shim/%s", urlPath)) + resp, err := client.Get(fmt.Sprintf("http://shim%s", urlPath)) if err != nil { return nil, err } @@ -65,15 +66,60 @@ func DoGet(sandboxID string, timeoutInSeconds time.Duration, urlPath string) ([] return body, nil } -func DoPost(sandboxID string, timeoutInSeconds time.Duration, urlPath string, payload []byte) error { +// DoPut will make a PUT request to the shim endpoint that handles the given sandbox ID +func DoPut(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentType string, payload []byte) error { client, err := BuildShimClient(sandboxID, timeoutInSeconds) if err != nil { return err } - resp, err := client.Post(fmt.Sprintf("http://shim/%s", urlPath), "application/json", bytes.NewBuffer(payload)) + req, err := http.NewRequest(http.MethodPut, fmt.Sprintf("http://shim%s", urlPath), bytes.NewBuffer(payload)) + if err != nil { + return err + } + req.Header.Set("Content-Type", contentType) + + resp, err := client.Do(req) + if err != nil { + return err + } + defer func() { - resp.Body.Close() + if resp != nil { + resp.Body.Close() + } }() - return err + + if resp.StatusCode != http.StatusOK { + data, _ := ioutil.ReadAll(resp.Body) + return fmt.Errorf("error sending put: url: %s, status code: %d, response data: %s", urlPath, resp.StatusCode, string(data)) + } + + return nil +} + +// DoPost will make a POST request to the shim endpoint that handles the given sandbox ID +func DoPost(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentType string, payload []byte) error { + client, err := BuildShimClient(sandboxID, timeoutInSeconds) + if err != nil { + return err + } + + resp, err := client.Post(fmt.Sprintf("http://shim%s", urlPath), contentType, bytes.NewBuffer(payload)) + if err != nil { + return err + } + + defer func() { + if resp != nil { + resp.Body.Close() + } + }() + + if resp.StatusCode != http.StatusOK { + data, _ := ioutil.ReadAll(resp.Body) + return fmt.Errorf("error sending post: url: %s, status code: %d, response data: %s", urlPath, resp.StatusCode, string(data)) + } + + return nil } diff --git a/src/runtime/virtcontainers/agent.go b/src/runtime/virtcontainers/agent.go index 94c9e14c0..25a1d73ff 100644 --- a/src/runtime/virtcontainers/agent.go +++ b/src/runtime/virtcontainers/agent.go @@ -191,7 +191,7 @@ type agent interface { // getAgentMetrics get metrics of agent and guest through agent getAgentMetrics(context.Context, *grpc.GetMetricsRequest) (*grpc.Metrics, error) - //getGuestVolumeStats get the filesystem stats of a volume specified by the volume mount path on the guest. + // getGuestVolumeStats get the filesystem stats of a volume specified by the volume mount path on the guest. getGuestVolumeStats(ctx context.Context, volumeGuestPath string) ([]byte, error) // resizeGuestVolume resizes a volume specified by the volume mount path on the guest. @@ -199,4 +199,9 @@ type agent interface { // pullImage will tell the agent to pull an image inside the Pod Sandbox image.ImageService + // getIPTables obtains the iptables from the guest + getIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) + + // setIPTables sets the iptables from the guest + setIPTables(ctx context.Context, isIPv6 bool, data []byte) error } diff --git a/src/runtime/virtcontainers/interfaces.go b/src/runtime/virtcontainers/interfaces.go index ec3fe87bf..272059fca 100644 --- a/src/runtime/virtcontainers/interfaces.go +++ b/src/runtime/virtcontainers/interfaces.go @@ -83,6 +83,8 @@ type VCSandbox interface { // Image management inside Sandbox image.ImageService + GetIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) + SetIPTables(ctx context.Context, isIPv6 bool, data []byte) error } // VCContainer is the Container interface diff --git a/src/runtime/virtcontainers/kata_agent.go b/src/runtime/virtcontainers/kata_agent.go index 62ac7a2dd..38fcae111 100644 --- a/src/runtime/virtcontainers/kata_agent.go +++ b/src/runtime/virtcontainers/kata_agent.go @@ -143,6 +143,8 @@ const ( grpcAddSwapRequest = "grpc.AddSwapRequest" grpcVolumeStatsRequest = "grpc.VolumeStatsRequest" grpcResizeVolumeRequest = "grpc.ResizeVolumeRequest" + grpcGetIPTablesRequest = "grpc.GetIPTablesRequest" + grpcSetIPTablesRequest = "grpc.SetIPTablesRequest" ) // newKataAgent returns an agent from an agent type. @@ -1981,6 +1983,12 @@ func (k *kataAgent) installReqFunc(c *kataclient.AgentClient) { k.reqHandlers[grpcResizeVolumeRequest] = func(ctx context.Context, req interface{}) (interface{}, error) { return k.client.AgentServiceClient.ResizeVolume(ctx, req.(*grpc.ResizeVolumeRequest)) } + k.reqHandlers[grpcGetIPTablesRequest] = func(ctx context.Context, req interface{}) (interface{}, error) { + return k.client.AgentServiceClient.GetIPTables(ctx, req.(*grpc.GetIPTablesRequest)) + } + k.reqHandlers[grpcSetIPTablesRequest] = func(ctx context.Context, req interface{}) (interface{}, error) { + return k.client.AgentServiceClient.SetIPTables(ctx, req.(*grpc.SetIPTablesRequest)) + } } func (k *kataAgent) getReqContext(ctx context.Context, reqName string) (newCtx context.Context, cancel context.CancelFunc) { @@ -2199,6 +2207,26 @@ func (k *kataAgent) getAgentMetrics(ctx context.Context, req *grpc.GetMetricsReq return resp.(*grpc.Metrics), nil } +func (k *kataAgent) getIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + resp, err := k.sendReq(ctx, &grpc.GetIPTablesRequest{IsIpv6: isIPv6}) + if err != nil { + return nil, err + } + return resp.(*grpc.GetIPTablesResponse).Data, nil +} + +func (k *kataAgent) setIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + _, err := k.sendReq(ctx, &grpc.SetIPTablesRequest{ + IsIpv6: isIPv6, + Data: data, + }) + if err != nil { + k.Logger().WithError(err).Errorf("setIPTables request to agent failed") + } + + return err +} + func (k *kataAgent) getGuestVolumeStats(ctx context.Context, volumeGuestPath string) ([]byte, error) { result, err := k.sendReq(ctx, &grpc.VolumeStatsRequest{VolumeGuestPath: volumeGuestPath}) if err != nil { diff --git a/src/runtime/virtcontainers/mock_agent.go b/src/runtime/virtcontainers/mock_agent.go index 4b22984a8..daff61d29 100644 --- a/src/runtime/virtcontainers/mock_agent.go +++ b/src/runtime/virtcontainers/mock_agent.go @@ -254,3 +254,11 @@ func (n *mockAgent) resizeGuestVolume(ctx context.Context, volumeGuestPath strin func (k *mockAgent) PullImage(ctx context.Context, req *image.PullImageReq) (*image.PullImageResp, error) { return nil, nil } + +func (k *mockAgent) getIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + return nil, nil +} + +func (k *mockAgent) setIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + return nil +} diff --git a/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go b/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go index 3a3e78504..64b19aa3b 100644 --- a/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go +++ b/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go @@ -1681,6 +1681,167 @@ func (m *AddARPNeighborsRequest) XXX_DiscardUnknown() { var xxx_messageInfo_AddARPNeighborsRequest proto.InternalMessageInfo +type GetIPTablesRequest struct { + IsIpv6 bool `protobuf:"varint,1,opt,name=is_ipv6,json=isIpv6,proto3" json:"is_ipv6,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *GetIPTablesRequest) Reset() { *m = GetIPTablesRequest{} } +func (*GetIPTablesRequest) ProtoMessage() {} +func (*GetIPTablesRequest) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{40} +} +func (m *GetIPTablesRequest) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *GetIPTablesRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_GetIPTablesRequest.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *GetIPTablesRequest) XXX_Merge(src proto.Message) { + xxx_messageInfo_GetIPTablesRequest.Merge(m, src) +} +func (m *GetIPTablesRequest) XXX_Size() int { + return m.Size() +} +func (m *GetIPTablesRequest) XXX_DiscardUnknown() { + xxx_messageInfo_GetIPTablesRequest.DiscardUnknown(m) +} + +var xxx_messageInfo_GetIPTablesRequest proto.InternalMessageInfo + +type GetIPTablesResponse struct { + // raw stdout from iptables-save or ip6tables-save + Data []byte `protobuf:"bytes,1,opt,name=data,proto3" json:"data,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *GetIPTablesResponse) Reset() { *m = GetIPTablesResponse{} } +func (*GetIPTablesResponse) ProtoMessage() {} +func (*GetIPTablesResponse) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{41} +} +func (m *GetIPTablesResponse) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *GetIPTablesResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_GetIPTablesResponse.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *GetIPTablesResponse) XXX_Merge(src proto.Message) { + xxx_messageInfo_GetIPTablesResponse.Merge(m, src) +} +func (m *GetIPTablesResponse) XXX_Size() int { + return m.Size() +} +func (m *GetIPTablesResponse) XXX_DiscardUnknown() { + xxx_messageInfo_GetIPTablesResponse.DiscardUnknown(m) +} + +var xxx_messageInfo_GetIPTablesResponse proto.InternalMessageInfo + +type SetIPTablesRequest struct { + IsIpv6 bool `protobuf:"varint,1,opt,name=is_ipv6,json=isIpv6,proto3" json:"is_ipv6,omitempty"` + // iptables, in raw format expected to be passed to stdin + // of iptables-save or ip6tables-save + Data []byte `protobuf:"bytes,2,opt,name=data,proto3" json:"data,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *SetIPTablesRequest) Reset() { *m = SetIPTablesRequest{} } +func (*SetIPTablesRequest) ProtoMessage() {} +func (*SetIPTablesRequest) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{42} +} +func (m *SetIPTablesRequest) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *SetIPTablesRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_SetIPTablesRequest.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *SetIPTablesRequest) XXX_Merge(src proto.Message) { + xxx_messageInfo_SetIPTablesRequest.Merge(m, src) +} +func (m *SetIPTablesRequest) XXX_Size() int { + return m.Size() +} +func (m *SetIPTablesRequest) XXX_DiscardUnknown() { + xxx_messageInfo_SetIPTablesRequest.DiscardUnknown(m) +} + +var xxx_messageInfo_SetIPTablesRequest proto.InternalMessageInfo + +type SetIPTablesResponse struct { + // raw stdout from iptables-restore or ip6tables-restore ? + Data []byte `protobuf:"bytes,1,opt,name=data,proto3" json:"data,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *SetIPTablesResponse) Reset() { *m = SetIPTablesResponse{} } +func (*SetIPTablesResponse) ProtoMessage() {} +func (*SetIPTablesResponse) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{43} +} +func (m *SetIPTablesResponse) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *SetIPTablesResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_SetIPTablesResponse.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *SetIPTablesResponse) XXX_Merge(src proto.Message) { + xxx_messageInfo_SetIPTablesResponse.Merge(m, src) +} +func (m *SetIPTablesResponse) XXX_Size() int { + return m.Size() +} +func (m *SetIPTablesResponse) XXX_DiscardUnknown() { + xxx_messageInfo_SetIPTablesResponse.DiscardUnknown(m) +} + +var xxx_messageInfo_SetIPTablesResponse proto.InternalMessageInfo + type OnlineCPUMemRequest struct { // Wait specifies if the caller waits for the agent to online all resources. // If true the agent returns once all resources have been connected, otherwise all @@ -1698,7 +1859,7 @@ type OnlineCPUMemRequest struct { func (m *OnlineCPUMemRequest) Reset() { *m = OnlineCPUMemRequest{} } func (*OnlineCPUMemRequest) ProtoMessage() {} func (*OnlineCPUMemRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{40} + return fileDescriptor_712ce9a559fda969, []int{44} } func (m *OnlineCPUMemRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1738,7 +1899,7 @@ type ReseedRandomDevRequest struct { func (m *ReseedRandomDevRequest) Reset() { *m = ReseedRandomDevRequest{} } func (*ReseedRandomDevRequest) ProtoMessage() {} func (*ReseedRandomDevRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{41} + return fileDescriptor_712ce9a559fda969, []int{45} } func (m *ReseedRandomDevRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1788,7 +1949,7 @@ type AgentDetails struct { func (m *AgentDetails) Reset() { *m = AgentDetails{} } func (*AgentDetails) ProtoMessage() {} func (*AgentDetails) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{42} + return fileDescriptor_712ce9a559fda969, []int{46} } func (m *AgentDetails) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1834,7 +1995,7 @@ type GuestDetailsRequest struct { func (m *GuestDetailsRequest) Reset() { *m = GuestDetailsRequest{} } func (*GuestDetailsRequest) ProtoMessage() {} func (*GuestDetailsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{43} + return fileDescriptor_712ce9a559fda969, []int{47} } func (m *GuestDetailsRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1876,7 +2037,7 @@ type GuestDetailsResponse struct { func (m *GuestDetailsResponse) Reset() { *m = GuestDetailsResponse{} } func (*GuestDetailsResponse) ProtoMessage() {} func (*GuestDetailsResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{44} + return fileDescriptor_712ce9a559fda969, []int{48} } func (m *GuestDetailsResponse) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1917,7 +2078,7 @@ type MemHotplugByProbeRequest struct { func (m *MemHotplugByProbeRequest) Reset() { *m = MemHotplugByProbeRequest{} } func (*MemHotplugByProbeRequest) ProtoMessage() {} func (*MemHotplugByProbeRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{45} + return fileDescriptor_712ce9a559fda969, []int{49} } func (m *MemHotplugByProbeRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1959,7 +2120,7 @@ type SetGuestDateTimeRequest struct { func (m *SetGuestDateTimeRequest) Reset() { *m = SetGuestDateTimeRequest{} } func (*SetGuestDateTimeRequest) ProtoMessage() {} func (*SetGuestDateTimeRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{46} + return fileDescriptor_712ce9a559fda969, []int{50} } func (m *SetGuestDateTimeRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2005,7 +2166,7 @@ type FSGroup struct { func (m *FSGroup) Reset() { *m = FSGroup{} } func (*FSGroup) ProtoMessage() {} func (*FSGroup) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{47} + return fileDescriptor_712ce9a559fda969, []int{51} } func (m *FSGroup) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2073,7 +2234,7 @@ type Storage struct { func (m *Storage) Reset() { *m = Storage{} } func (*Storage) ProtoMessage() {} func (*Storage) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{48} + return fileDescriptor_712ce9a559fda969, []int{52} } func (m *Storage) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2144,7 +2305,7 @@ type Device struct { func (m *Device) Reset() { *m = Device{} } func (*Device) ProtoMessage() {} func (*Device) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{49} + return fileDescriptor_712ce9a559fda969, []int{53} } func (m *Device) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2185,7 +2346,7 @@ type StringUser struct { func (m *StringUser) Reset() { *m = StringUser{} } func (*StringUser) ProtoMessage() {} func (*StringUser) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{50} + return fileDescriptor_712ce9a559fda969, []int{54} } func (m *StringUser) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2242,7 +2403,7 @@ type CopyFileRequest struct { func (m *CopyFileRequest) Reset() { *m = CopyFileRequest{} } func (*CopyFileRequest) ProtoMessage() {} func (*CopyFileRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{51} + return fileDescriptor_712ce9a559fda969, []int{55} } func (m *CopyFileRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2280,7 +2441,7 @@ type GetOOMEventRequest struct { func (m *GetOOMEventRequest) Reset() { *m = GetOOMEventRequest{} } func (*GetOOMEventRequest) ProtoMessage() {} func (*GetOOMEventRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{52} + return fileDescriptor_712ce9a559fda969, []int{56} } func (m *GetOOMEventRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2319,7 +2480,7 @@ type OOMEvent struct { func (m *OOMEvent) Reset() { *m = OOMEvent{} } func (*OOMEvent) ProtoMessage() {} func (*OOMEvent) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{53} + return fileDescriptor_712ce9a559fda969, []int{57} } func (m *OOMEvent) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2358,7 +2519,7 @@ type AddSwapRequest struct { func (m *AddSwapRequest) Reset() { *m = AddSwapRequest{} } func (*AddSwapRequest) ProtoMessage() {} func (*AddSwapRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{54} + return fileDescriptor_712ce9a559fda969, []int{58} } func (m *AddSwapRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2396,7 +2557,7 @@ type GetMetricsRequest struct { func (m *GetMetricsRequest) Reset() { *m = GetMetricsRequest{} } func (*GetMetricsRequest) ProtoMessage() {} func (*GetMetricsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{55} + return fileDescriptor_712ce9a559fda969, []int{59} } func (m *GetMetricsRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2435,7 +2596,7 @@ type Metrics struct { func (m *Metrics) Reset() { *m = Metrics{} } func (*Metrics) ProtoMessage() {} func (*Metrics) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{56} + return fileDescriptor_712ce9a559fda969, []int{60} } func (m *Metrics) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2475,7 +2636,7 @@ type VolumeStatsRequest struct { func (m *VolumeStatsRequest) Reset() { *m = VolumeStatsRequest{} } func (*VolumeStatsRequest) ProtoMessage() {} func (*VolumeStatsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{57} + return fileDescriptor_712ce9a559fda969, []int{61} } func (m *VolumeStatsRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2516,7 +2677,7 @@ type ResizeVolumeRequest struct { func (m *ResizeVolumeRequest) Reset() { *m = ResizeVolumeRequest{} } func (*ResizeVolumeRequest) ProtoMessage() {} func (*ResizeVolumeRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{58} + return fileDescriptor_712ce9a559fda969, []int{62} } func (m *ResizeVolumeRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2588,6 +2749,10 @@ func init() { proto.RegisterType((*ListRoutesRequest)(nil), "grpc.ListRoutesRequest") proto.RegisterType((*ARPNeighbors)(nil), "grpc.ARPNeighbors") proto.RegisterType((*AddARPNeighborsRequest)(nil), "grpc.AddARPNeighborsRequest") + proto.RegisterType((*GetIPTablesRequest)(nil), "grpc.GetIPTablesRequest") + proto.RegisterType((*GetIPTablesResponse)(nil), "grpc.GetIPTablesResponse") + proto.RegisterType((*SetIPTablesRequest)(nil), "grpc.SetIPTablesRequest") + proto.RegisterType((*SetIPTablesResponse)(nil), "grpc.SetIPTablesResponse") proto.RegisterType((*OnlineCPUMemRequest)(nil), "grpc.OnlineCPUMemRequest") proto.RegisterType((*ReseedRandomDevRequest)(nil), "grpc.ReseedRandomDevRequest") proto.RegisterType((*AgentDetails)(nil), "grpc.AgentDetails") @@ -2614,203 +2779,208 @@ func init() { } var fileDescriptor_712ce9a559fda969 = []byte{ - // 3127 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x1a, 0xcb, 0x72, 0x24, 0x47, - 0xd1, 0xf3, 0x90, 0x66, 0x26, 0xe7, 0xa5, 0x69, 0x69, 0xb5, 0xb3, 0x63, 0x5b, 0xac, 0x7b, 0xed, - 0xb5, 0x6c, 0x63, 0xc9, 0x5e, 0x3b, 0x58, 0x3f, 0xc2, 0x2c, 0x92, 0x56, 0x96, 0x64, 0x5b, 0xde, - 0xa1, 0x65, 0x61, 0x02, 0x02, 0x3a, 0x7a, 0xba, 0x4b, 0x33, 0x65, 0x4d, 0x77, 0xb5, 0xab, 0xab, - 0xb5, 0x1a, 0x13, 0x41, 0x70, 0x82, 0x1b, 0x47, 0x6e, 0xfc, 0x00, 0xc1, 0x1f, 0x70, 0xe1, 0xc0, - 0xc1, 0xc1, 0x89, 0x23, 0x17, 0x08, 0xbc, 0x9f, 0xc0, 0x17, 0x10, 0xf5, 0xea, 0xc7, 0x3c, 0x64, - 0x50, 0x6c, 0x04, 0x97, 0x89, 0xce, 0xac, 0xac, 0x7c, 0x55, 0x65, 0x56, 0x66, 0xd5, 0x40, 0x7f, - 0x88, 0xd9, 0x28, 0x1e, 0x6c, 0xb9, 0xc4, 0xdf, 0x3e, 0x77, 0x98, 0xf3, 0xba, 0x4b, 0x02, 0xe6, - 0xe0, 0x00, 0xd1, 0x68, 0x06, 0x8e, 0xa8, 0xbb, 0x3d, 0xc6, 0x83, 0x68, 0x3b, 0xa4, 0x84, 0x11, - 0x97, 0x8c, 0xd5, 0x57, 0xb4, 0xed, 0x0c, 0x51, 0xc0, 0xb6, 0x04, 0x60, 0x94, 0x87, 0x34, 0x74, - 0x7b, 0x35, 0xe2, 0x62, 0x89, 0xe8, 0xd5, 0xdc, 0x48, 0x7f, 0xd6, 0xd9, 0x24, 0x44, 0x91, 0x02, - 0x9e, 0x1d, 0x12, 0x32, 0x1c, 0x23, 0xc9, 0x63, 0x10, 0x9f, 0x6d, 0x23, 0x3f, 0x64, 0x13, 0x39, - 0x68, 0xfe, 0xbe, 0x08, 0xeb, 0x7b, 0x14, 0x39, 0x0c, 0xed, 0x69, 0x05, 0x2c, 0xf4, 0x65, 0x8c, - 0x22, 0x66, 0xbc, 0x00, 0x8d, 0x44, 0x29, 0x1b, 0x7b, 0xdd, 0xc2, 0xed, 0xc2, 0x66, 0xcd, 0xaa, - 0x27, 0xb8, 0x23, 0xcf, 0xb8, 0x09, 0x15, 0x74, 0x89, 0x5c, 0x3e, 0x5a, 0x14, 0xa3, 0xcb, 0x1c, - 0x3c, 0xf2, 0x8c, 0x37, 0xa1, 0x1e, 0x31, 0x8a, 0x83, 0xa1, 0x1d, 0x47, 0x88, 0x76, 0x4b, 0xb7, - 0x0b, 0x9b, 0xf5, 0x7b, 0x2b, 0x5b, 0x5c, 0xe5, 0xad, 0x13, 0x31, 0x70, 0x1a, 0x21, 0x6a, 0x41, - 0x94, 0x7c, 0x1b, 0x77, 0xa1, 0xe2, 0xa1, 0x0b, 0xec, 0xa2, 0xa8, 0x5b, 0xbe, 0x5d, 0xda, 0xac, - 0xdf, 0x6b, 0x48, 0xf2, 0x87, 0x02, 0x69, 0xe9, 0x41, 0xe3, 0x15, 0xa8, 0x46, 0x8c, 0x50, 0x67, - 0x88, 0xa2, 0xee, 0x92, 0x20, 0x6c, 0x6a, 0xbe, 0x02, 0x6b, 0x25, 0xc3, 0xc6, 0x73, 0x50, 0x7a, - 0xb4, 0x77, 0xd4, 0x5d, 0x16, 0xd2, 0x41, 0x51, 0x85, 0xc8, 0xb5, 0x38, 0xda, 0xb8, 0x03, 0xcd, - 0xc8, 0x09, 0xbc, 0x01, 0xb9, 0xb4, 0x43, 0xec, 0x05, 0x51, 0xb7, 0x72, 0xbb, 0xb0, 0x59, 0xb5, - 0x1a, 0x0a, 0xd9, 0xe7, 0x38, 0xf3, 0x3d, 0xb8, 0x71, 0xc2, 0x1c, 0xca, 0xae, 0xe1, 0x1d, 0xf3, - 0x14, 0xd6, 0x2d, 0xe4, 0x93, 0x8b, 0x6b, 0xb9, 0xb6, 0x0b, 0x15, 0x86, 0x7d, 0x44, 0x62, 0x26, - 0x5c, 0xdb, 0xb4, 0x34, 0x68, 0xfe, 0xb1, 0x00, 0xc6, 0xfe, 0x25, 0x72, 0xfb, 0x94, 0xb8, 0x28, - 0x8a, 0xfe, 0x4f, 0xcb, 0xf5, 0x32, 0x54, 0x42, 0xa9, 0x40, 0xb7, 0x2c, 0xc8, 0xd5, 0x2a, 0x68, - 0xad, 0xf4, 0xa8, 0xf9, 0x05, 0xac, 0x9d, 0xe0, 0x61, 0xe0, 0x8c, 0x9f, 0xa2, 0xbe, 0xeb, 0xb0, - 0x1c, 0x09, 0x9e, 0x42, 0xd5, 0xa6, 0xa5, 0x20, 0xb3, 0x0f, 0xc6, 0xe7, 0x0e, 0x66, 0x4f, 0x4f, - 0x92, 0xf9, 0x3a, 0xac, 0xe6, 0x38, 0x46, 0x21, 0x09, 0x22, 0x24, 0x14, 0x60, 0x0e, 0x8b, 0x23, - 0xc1, 0x6c, 0xc9, 0x52, 0x90, 0x49, 0x60, 0xfd, 0x34, 0xf4, 0xae, 0x19, 0x4d, 0xf7, 0xa0, 0x46, - 0x51, 0x44, 0x62, 0xca, 0x63, 0xa0, 0x28, 0x9c, 0xba, 0x26, 0x9d, 0xfa, 0x09, 0x0e, 0xe2, 0x4b, - 0x4b, 0x8f, 0x59, 0x29, 0x99, 0xda, 0x9f, 0x2c, 0xba, 0xce, 0xfe, 0x7c, 0x0f, 0x6e, 0xf4, 0x9d, - 0x38, 0xba, 0x8e, 0xae, 0xe6, 0xfb, 0x7c, 0x6f, 0x47, 0xb1, 0x7f, 0xad, 0xc9, 0x7f, 0x28, 0x40, - 0x75, 0x2f, 0x8c, 0x4f, 0x23, 0x67, 0x88, 0x8c, 0xef, 0x40, 0x9d, 0x11, 0xe6, 0x8c, 0xed, 0x98, - 0x83, 0x82, 0xbc, 0x6c, 0x81, 0x40, 0x49, 0x82, 0x17, 0xa0, 0x11, 0x22, 0xea, 0x86, 0xb1, 0xa2, - 0x28, 0xde, 0x2e, 0x6d, 0x96, 0xad, 0xba, 0xc4, 0x49, 0x92, 0x2d, 0x58, 0x15, 0x63, 0x36, 0x0e, - 0xec, 0x73, 0x44, 0x03, 0x34, 0xf6, 0x89, 0x87, 0xc4, 0xe6, 0x28, 0x5b, 0x1d, 0x31, 0x74, 0x14, - 0x7c, 0x9c, 0x0c, 0x18, 0xaf, 0x42, 0x27, 0xa1, 0xe7, 0x3b, 0x5e, 0x50, 0x97, 0x05, 0x75, 0x5b, - 0x51, 0x9f, 0x2a, 0xb4, 0xf9, 0x4b, 0x68, 0x7d, 0x36, 0xa2, 0x84, 0xb1, 0x31, 0x0e, 0x86, 0x0f, - 0x1d, 0xe6, 0xf0, 0xd0, 0x0c, 0x11, 0xc5, 0xc4, 0x8b, 0x94, 0xb6, 0x1a, 0x34, 0x5e, 0x83, 0x0e, - 0x93, 0xb4, 0xc8, 0xb3, 0x35, 0x4d, 0x51, 0xd0, 0xac, 0x24, 0x03, 0x7d, 0x45, 0xfc, 0x12, 0xb4, - 0x52, 0x62, 0x1e, 0xdc, 0x4a, 0xdf, 0x66, 0x82, 0xfd, 0x0c, 0xfb, 0xc8, 0xbc, 0x10, 0xbe, 0x12, - 0x8b, 0x6c, 0xbc, 0x06, 0xb5, 0xd4, 0x0f, 0x05, 0xb1, 0x43, 0x5a, 0x72, 0x87, 0x68, 0x77, 0x5a, - 0xd5, 0xc4, 0x29, 0x1f, 0x40, 0x9b, 0x25, 0x8a, 0xdb, 0x9e, 0xc3, 0x9c, 0xfc, 0xa6, 0xca, 0x5b, - 0x65, 0xb5, 0x58, 0x0e, 0x36, 0xdf, 0x87, 0x5a, 0x1f, 0x7b, 0x91, 0x14, 0xdc, 0x85, 0x8a, 0x1b, - 0x53, 0x8a, 0x02, 0xa6, 0x4d, 0x56, 0xa0, 0xb1, 0x06, 0x4b, 0x63, 0xec, 0x63, 0xa6, 0xcc, 0x94, - 0x80, 0x49, 0x00, 0x8e, 0x91, 0x4f, 0xe8, 0x44, 0x38, 0x6c, 0x0d, 0x96, 0xb2, 0x8b, 0x2b, 0x01, - 0xe3, 0x59, 0xa8, 0xf9, 0xce, 0x65, 0xb2, 0xa8, 0x7c, 0xa4, 0xea, 0x3b, 0x97, 0x52, 0xf9, 0x2e, - 0x54, 0xce, 0x1c, 0x3c, 0x76, 0x03, 0xa6, 0xbc, 0xa2, 0xc1, 0x54, 0x60, 0x39, 0x2b, 0xf0, 0x2f, - 0x45, 0xa8, 0x4b, 0x89, 0x52, 0xe1, 0x35, 0x58, 0x72, 0x1d, 0x77, 0x94, 0x88, 0x14, 0x80, 0x71, - 0x57, 0x2b, 0x52, 0xcc, 0x66, 0xb8, 0x54, 0x53, 0xad, 0xda, 0x36, 0x40, 0xf4, 0xd8, 0x09, 0x95, - 0x6e, 0xa5, 0x05, 0xc4, 0x35, 0x4e, 0x23, 0xd5, 0x7d, 0x0b, 0x1a, 0x72, 0xdf, 0xa9, 0x29, 0xe5, - 0x05, 0x53, 0xea, 0x92, 0x4a, 0x4e, 0xba, 0x03, 0xcd, 0x38, 0x42, 0xf6, 0x08, 0x23, 0xea, 0x50, - 0x77, 0x34, 0xe9, 0x2e, 0xc9, 0x03, 0x28, 0x8e, 0xd0, 0xa1, 0xc6, 0x19, 0xf7, 0x60, 0x89, 0xe7, - 0x96, 0xa8, 0xbb, 0x2c, 0xce, 0xba, 0xe7, 0xb2, 0x2c, 0x85, 0xa9, 0x5b, 0xe2, 0x77, 0x3f, 0x60, - 0x74, 0x62, 0x49, 0xd2, 0xde, 0x3b, 0x00, 0x29, 0xd2, 0x58, 0x81, 0xd2, 0x39, 0x9a, 0xa8, 0x38, - 0xe4, 0x9f, 0xdc, 0x39, 0x17, 0xce, 0x38, 0xd6, 0x5e, 0x97, 0xc0, 0x7b, 0xc5, 0x77, 0x0a, 0xa6, - 0x0b, 0xed, 0xdd, 0xf1, 0x39, 0x26, 0x99, 0xe9, 0x6b, 0xb0, 0xe4, 0x3b, 0x5f, 0x10, 0xaa, 0x3d, - 0x29, 0x00, 0x81, 0xc5, 0x01, 0xa1, 0x9a, 0x85, 0x00, 0x8c, 0x16, 0x14, 0x49, 0x28, 0xfc, 0x55, - 0xb3, 0x8a, 0x24, 0x4c, 0x05, 0x95, 0x33, 0x82, 0xcc, 0x7f, 0x96, 0x01, 0x52, 0x29, 0x86, 0x05, - 0x3d, 0x4c, 0xec, 0x08, 0x51, 0x7e, 0xbe, 0xdb, 0x83, 0x09, 0x43, 0x91, 0x4d, 0x91, 0x1b, 0xd3, - 0x08, 0x5f, 0xf0, 0xf5, 0xe3, 0x66, 0xdf, 0x90, 0x66, 0x4f, 0xe9, 0x66, 0xdd, 0xc4, 0xe4, 0x44, - 0xce, 0xdb, 0xe5, 0xd3, 0x2c, 0x3d, 0xcb, 0x38, 0x82, 0x1b, 0x29, 0x4f, 0x2f, 0xc3, 0xae, 0x78, - 0x15, 0xbb, 0xd5, 0x84, 0x9d, 0x97, 0xb2, 0xda, 0x87, 0x55, 0x4c, 0xec, 0x2f, 0x63, 0x14, 0xe7, - 0x18, 0x95, 0xae, 0x62, 0xd4, 0xc1, 0xe4, 0x87, 0x62, 0x42, 0xca, 0xa6, 0x0f, 0xb7, 0x32, 0x56, - 0xf2, 0x70, 0xcf, 0x30, 0x2b, 0x5f, 0xc5, 0x6c, 0x3d, 0xd1, 0x8a, 0xe7, 0x83, 0x94, 0xe3, 0x47, - 0xb0, 0x8e, 0x89, 0xfd, 0xd8, 0xc1, 0x6c, 0x9a, 0xdd, 0xd2, 0xb7, 0x18, 0xc9, 0x4f, 0xb4, 0x3c, - 0x2f, 0x69, 0xa4, 0x8f, 0xe8, 0x30, 0x67, 0xe4, 0xf2, 0xb7, 0x18, 0x79, 0x2c, 0x26, 0xa4, 0x6c, - 0x76, 0xa0, 0x83, 0xc9, 0xb4, 0x36, 0x95, 0xab, 0x98, 0xb4, 0x31, 0xc9, 0x6b, 0xb2, 0x0b, 0x9d, - 0x08, 0xb9, 0x8c, 0xd0, 0xec, 0x26, 0xa8, 0x5e, 0xc5, 0x62, 0x45, 0xd1, 0x27, 0x3c, 0xcc, 0x9f, - 0x42, 0xe3, 0x30, 0x1e, 0x22, 0x36, 0x1e, 0x24, 0xc9, 0xe0, 0xa9, 0xe5, 0x1f, 0xf3, 0xdf, 0x45, - 0xa8, 0xef, 0x0d, 0x29, 0x89, 0xc3, 0x5c, 0x4e, 0x96, 0x41, 0x3a, 0x9d, 0x93, 0x05, 0x89, 0xc8, - 0xc9, 0x92, 0xf8, 0x6d, 0x68, 0xf8, 0x22, 0x74, 0x15, 0xbd, 0xcc, 0x43, 0x9d, 0x99, 0xa0, 0xb6, - 0xea, 0x7e, 0x26, 0x99, 0x6d, 0x01, 0x84, 0xd8, 0x8b, 0xd4, 0x1c, 0x99, 0x8e, 0xda, 0xaa, 0xdc, - 0xd2, 0x29, 0xda, 0xaa, 0x85, 0x49, 0xb6, 0x7e, 0x13, 0xea, 0x03, 0xee, 0x24, 0x35, 0x21, 0x97, - 0x8c, 0x52, 0xef, 0x59, 0x30, 0x48, 0x83, 0xf0, 0x10, 0x9a, 0x23, 0xe9, 0x32, 0x35, 0x49, 0xee, - 0xa1, 0x3b, 0xca, 0x92, 0xd4, 0xde, 0xad, 0xac, 0x67, 0xe5, 0x02, 0x34, 0x46, 0x19, 0x54, 0xef, - 0x04, 0x3a, 0x33, 0x24, 0x73, 0x72, 0xd0, 0x66, 0x36, 0x07, 0xd5, 0xef, 0x19, 0x52, 0x50, 0x76, - 0x66, 0x36, 0x2f, 0xfd, 0xb6, 0x08, 0x8d, 0x4f, 0x11, 0x7b, 0x4c, 0xe8, 0xb9, 0xd4, 0xd7, 0x80, - 0x72, 0xe0, 0xf8, 0x48, 0x71, 0x14, 0xdf, 0xc6, 0x2d, 0xa8, 0xd2, 0x4b, 0x99, 0x40, 0xd4, 0x7a, - 0x56, 0xe8, 0xa5, 0x48, 0x0c, 0xc6, 0xf3, 0x00, 0xf4, 0xd2, 0x0e, 0x1d, 0xf7, 0x1c, 0x29, 0x0f, - 0x96, 0xad, 0x1a, 0xbd, 0xec, 0x4b, 0x04, 0xdf, 0x0a, 0xf4, 0xd2, 0x46, 0x94, 0x12, 0x1a, 0xa9, - 0x5c, 0x55, 0xa5, 0x97, 0xfb, 0x02, 0x56, 0x73, 0x3d, 0x4a, 0xc2, 0x10, 0x79, 0x22, 0x47, 0x8b, - 0xb9, 0x0f, 0x25, 0x82, 0x4b, 0x65, 0x5a, 0xea, 0xb2, 0x94, 0xca, 0x52, 0xa9, 0x2c, 0x95, 0x5a, - 0x91, 0x33, 0x59, 0x56, 0x2a, 0x4b, 0xa4, 0x56, 0xa5, 0x54, 0x96, 0x91, 0xca, 0x52, 0xa9, 0x35, - 0x3d, 0x57, 0x49, 0x35, 0x7f, 0x53, 0x80, 0xf5, 0xe9, 0xc2, 0x4f, 0xd5, 0xa6, 0x6f, 0x43, 0xc3, - 0x15, 0xeb, 0x95, 0xdb, 0x93, 0x9d, 0x99, 0x95, 0xb4, 0xea, 0x6e, 0x66, 0x1b, 0xdf, 0x87, 0x66, - 0x20, 0x1d, 0x9c, 0x6c, 0xcd, 0x52, 0xba, 0x2e, 0x59, 0xdf, 0x5b, 0x8d, 0x20, 0x03, 0x99, 0x1e, - 0x18, 0x9f, 0x53, 0xcc, 0xd0, 0x09, 0xa3, 0xc8, 0xf1, 0x9f, 0x46, 0x75, 0x6f, 0x40, 0x59, 0x54, - 0x2b, 0x7c, 0x99, 0x1a, 0x96, 0xf8, 0x36, 0x5f, 0x86, 0xd5, 0x9c, 0x14, 0x65, 0xeb, 0x0a, 0x94, - 0xc6, 0x28, 0x10, 0xdc, 0x9b, 0x16, 0xff, 0x34, 0x1d, 0xe8, 0x58, 0xc8, 0xf1, 0x9e, 0x9e, 0x36, - 0x4a, 0x44, 0x29, 0x15, 0xb1, 0x09, 0x46, 0x56, 0x84, 0x52, 0x45, 0x6b, 0x5d, 0xc8, 0x68, 0xfd, - 0x08, 0x3a, 0x7b, 0x63, 0x12, 0xa1, 0x13, 0xe6, 0xe1, 0xe0, 0x69, 0xb4, 0x23, 0xbf, 0x80, 0xd5, - 0xcf, 0xd8, 0xe4, 0x73, 0xce, 0x2c, 0xc2, 0x5f, 0xa1, 0xa7, 0x64, 0x1f, 0x25, 0x8f, 0xb5, 0x7d, - 0x94, 0x3c, 0xe6, 0xcd, 0x8d, 0x4b, 0xc6, 0xb1, 0x1f, 0x88, 0x50, 0x68, 0x5a, 0x0a, 0x32, 0x77, - 0xa1, 0x21, 0x6b, 0xe8, 0x63, 0xe2, 0xc5, 0x63, 0x34, 0x37, 0x06, 0x37, 0x00, 0x42, 0x87, 0x3a, - 0x3e, 0x62, 0x88, 0xca, 0x3d, 0x54, 0xb3, 0x32, 0x18, 0xf3, 0x77, 0x45, 0x58, 0x93, 0xf7, 0x0d, - 0x27, 0xb2, 0xcd, 0xd6, 0x26, 0xf4, 0xa0, 0x3a, 0x22, 0x11, 0xcb, 0x30, 0x4c, 0x60, 0xae, 0x22, - 0xef, 0xcf, 0x25, 0x37, 0xfe, 0x99, 0xbb, 0x04, 0x28, 0x5d, 0x7d, 0x09, 0x30, 0xd3, 0xe6, 0x97, - 0x67, 0xdb, 0x7c, 0x1e, 0x6d, 0x9a, 0x08, 0xcb, 0x18, 0xaf, 0x59, 0x35, 0x85, 0x39, 0xf2, 0x8c, - 0xbb, 0xd0, 0x1e, 0x72, 0x2d, 0xed, 0x11, 0x21, 0xe7, 0x76, 0xe8, 0xb0, 0x91, 0x08, 0xf5, 0x9a, - 0xd5, 0x14, 0xe8, 0x43, 0x42, 0xce, 0xfb, 0x0e, 0x1b, 0x19, 0xef, 0x42, 0x4b, 0x95, 0x81, 0xbe, - 0x70, 0x51, 0xa4, 0x0e, 0x3f, 0x15, 0x45, 0x59, 0xef, 0x59, 0xcd, 0xf3, 0x0c, 0x14, 0x99, 0x37, - 0xe1, 0xc6, 0x43, 0x14, 0x31, 0x4a, 0x26, 0x79, 0xc7, 0x98, 0xdf, 0x07, 0x38, 0x0a, 0x18, 0xa2, - 0x67, 0x8e, 0x8b, 0x22, 0xe3, 0x8d, 0x2c, 0xa4, 0x8a, 0xa3, 0x95, 0x2d, 0x79, 0xdd, 0x93, 0x0c, - 0x58, 0x19, 0x1a, 0x73, 0x0b, 0x96, 0x2d, 0x12, 0xf3, 0x74, 0xf4, 0xa2, 0xfe, 0x52, 0xf3, 0x1a, - 0x6a, 0x9e, 0x40, 0x5a, 0x6a, 0xcc, 0x3c, 0xd4, 0x2d, 0x6c, 0xca, 0x4e, 0x2d, 0xd1, 0x16, 0xd4, - 0xb0, 0xc6, 0xa9, 0xac, 0x32, 0x2b, 0x3a, 0x25, 0x31, 0xdf, 0x87, 0x55, 0xc9, 0x49, 0x72, 0xd6, - 0x6c, 0x5e, 0x84, 0x65, 0xaa, 0xd5, 0x28, 0xa4, 0xf7, 0x3c, 0x8a, 0x48, 0x8d, 0x71, 0x7f, 0x7c, - 0x82, 0x23, 0x96, 0x1a, 0xa2, 0xfd, 0xb1, 0x0a, 0x1d, 0x3e, 0x90, 0xe3, 0x69, 0x7e, 0x08, 0x8d, - 0x1d, 0xab, 0xff, 0x29, 0xc2, 0xc3, 0xd1, 0x80, 0x67, 0xcf, 0xef, 0xe5, 0x61, 0x65, 0xb0, 0xa1, - 0xb4, 0xcd, 0x0c, 0x59, 0x39, 0x3a, 0xf3, 0x23, 0x58, 0xdf, 0xf1, 0xbc, 0x2c, 0x4a, 0x6b, 0xfd, - 0x06, 0xd4, 0x82, 0x0c, 0xbb, 0xcc, 0x99, 0x95, 0xa3, 0x4e, 0x89, 0xcc, 0x9f, 0xc1, 0xea, 0xa3, - 0x60, 0x8c, 0x03, 0xb4, 0xd7, 0x3f, 0x3d, 0x46, 0x49, 0x2e, 0x32, 0xa0, 0xcc, 0x6b, 0x36, 0xc1, - 0xa3, 0x6a, 0x89, 0x6f, 0x1e, 0x9c, 0xc1, 0xc0, 0x76, 0xc3, 0x38, 0x52, 0x97, 0x3d, 0xcb, 0xc1, - 0x60, 0x2f, 0x8c, 0x23, 0x7e, 0xb8, 0xf0, 0xe2, 0x82, 0x04, 0xe3, 0x89, 0x88, 0xd0, 0xaa, 0x55, - 0x71, 0xc3, 0xf8, 0x51, 0x30, 0x9e, 0x98, 0xdf, 0x15, 0x1d, 0x38, 0x42, 0x9e, 0xe5, 0x04, 0x1e, - 0xf1, 0x1f, 0xa2, 0x8b, 0x8c, 0x84, 0xa4, 0xdb, 0xd3, 0x99, 0xe8, 0xeb, 0x02, 0x34, 0x76, 0x86, - 0x28, 0x60, 0x0f, 0x11, 0x73, 0xf0, 0x58, 0x74, 0x74, 0x17, 0x88, 0x46, 0x98, 0x04, 0x2a, 0xdc, - 0x34, 0xc8, 0x1b, 0x72, 0x1c, 0x60, 0x66, 0x7b, 0x0e, 0xf2, 0x49, 0x20, 0xb8, 0x54, 0x2d, 0xe0, - 0xa8, 0x87, 0x02, 0x63, 0xbc, 0x0c, 0x6d, 0x79, 0x19, 0x67, 0x8f, 0x9c, 0xc0, 0x1b, 0xf3, 0x40, - 0x2f, 0x89, 0xd0, 0x6c, 0x49, 0xf4, 0xa1, 0xc2, 0x1a, 0xaf, 0xc0, 0x8a, 0x0a, 0xc3, 0x94, 0xb2, - 0x2c, 0x28, 0xdb, 0x0a, 0x9f, 0x23, 0x8d, 0xc3, 0x90, 0x50, 0x16, 0xd9, 0x11, 0x72, 0x5d, 0xe2, - 0x87, 0xaa, 0x1d, 0x6a, 0x6b, 0xfc, 0x89, 0x44, 0x9b, 0x43, 0x58, 0x3d, 0xe0, 0x76, 0x2a, 0x4b, - 0xd2, 0x6d, 0xd5, 0xf2, 0x91, 0x6f, 0x0f, 0xc6, 0xc4, 0x3d, 0xb7, 0x79, 0x72, 0x54, 0x1e, 0xe6, - 0x05, 0xd7, 0x2e, 0x47, 0x9e, 0xe0, 0xaf, 0x44, 0xe7, 0xcf, 0xa9, 0x46, 0x84, 0x85, 0xe3, 0x78, - 0x68, 0x87, 0x94, 0x0c, 0x90, 0x32, 0xb1, 0xed, 0x23, 0xff, 0x50, 0xe2, 0xfb, 0x1c, 0x6d, 0xfe, - 0xa9, 0x00, 0x6b, 0x79, 0x49, 0x2a, 0xd5, 0x6f, 0xc3, 0x5a, 0x5e, 0x94, 0x3a, 0xfe, 0x65, 0x79, - 0xd9, 0xc9, 0x0a, 0x94, 0x85, 0xc0, 0x7d, 0x68, 0x8a, 0xab, 0x5b, 0xdb, 0x93, 0x9c, 0xf2, 0x45, - 0x4f, 0x76, 0x5d, 0xac, 0x86, 0x93, 0x5d, 0xa5, 0x77, 0xe1, 0x96, 0x32, 0xdf, 0x9e, 0x55, 0x5b, - 0x6e, 0x88, 0x75, 0x45, 0x70, 0x3c, 0xa5, 0xfd, 0x27, 0xd0, 0x4d, 0x51, 0xbb, 0x13, 0x81, 0x4c, - 0x37, 0xf3, 0xea, 0x94, 0xb1, 0x3b, 0x9e, 0x47, 0x45, 0x94, 0x94, 0xad, 0x79, 0x43, 0xe6, 0x03, - 0xb8, 0x79, 0x82, 0x98, 0xf4, 0x86, 0xc3, 0x54, 0x27, 0x22, 0x99, 0xad, 0x40, 0xe9, 0x04, 0xb9, - 0xc2, 0xf8, 0x92, 0xc5, 0x3f, 0xf9, 0x06, 0x3c, 0x8d, 0x90, 0x2b, 0xac, 0x2c, 0x59, 0xe2, 0xdb, - 0x0c, 0xa1, 0xf2, 0xe1, 0xc9, 0x01, 0xaf, 0x37, 0xf8, 0xa6, 0x96, 0xf5, 0x89, 0x3a, 0x8b, 0x9a, - 0x56, 0x45, 0xc0, 0x47, 0x9e, 0xf1, 0x11, 0xac, 0xca, 0x21, 0x77, 0xe4, 0x04, 0x43, 0x64, 0x87, - 0x64, 0x8c, 0x5d, 0xb9, 0xf5, 0x5b, 0xf7, 0x7a, 0x2a, 0x7c, 0x15, 0x9f, 0x3d, 0x41, 0xd2, 0x17, - 0x14, 0x56, 0x67, 0x38, 0x8d, 0x32, 0xff, 0x51, 0x80, 0x8a, 0x3a, 0x0e, 0xf8, 0x91, 0xe6, 0x51, - 0x7c, 0x81, 0xa8, 0xda, 0xec, 0x0a, 0x32, 0x5e, 0x82, 0x96, 0xfc, 0xb2, 0x49, 0xc8, 0x30, 0x49, - 0x0e, 0x99, 0xa6, 0xc4, 0x3e, 0x92, 0x48, 0x71, 0xdd, 0x27, 0x2e, 0xdc, 0x54, 0x6f, 0xab, 0x20, - 0x8e, 0x3f, 0x8b, 0xb8, 0x52, 0xe2, 0x50, 0xa9, 0x59, 0x0a, 0xe2, 0xc1, 0xa5, 0xf9, 0x2d, 0x09, - 0x7e, 0x1a, 0xe4, 0xc1, 0xe5, 0x93, 0x38, 0x60, 0x76, 0x48, 0x70, 0xc0, 0xd4, 0x29, 0x02, 0x02, - 0xd5, 0xe7, 0x18, 0x63, 0x13, 0xaa, 0x67, 0x91, 0x2d, 0xac, 0x11, 0x15, 0x63, 0x72, 0xb2, 0x29, - 0xab, 0xad, 0xca, 0x59, 0x24, 0x3e, 0xcc, 0x5f, 0x17, 0x60, 0x59, 0x5e, 0x8e, 0xf3, 0xbe, 0x3b, - 0x39, 0xf5, 0x8b, 0x58, 0x54, 0x50, 0x42, 0x2b, 0x79, 0xd2, 0x8b, 0x6f, 0x9e, 0x63, 0x2e, 0x7c, - 0x79, 0x76, 0x29, 0x23, 0x2e, 0x7c, 0x71, 0x68, 0xbd, 0x04, 0xad, 0xb4, 0x78, 0x10, 0xe3, 0xd2, - 0x98, 0x66, 0x82, 0x15, 0x64, 0x0b, 0x6d, 0x32, 0x7f, 0x0c, 0x90, 0x5e, 0x12, 0xf3, 0xed, 0x10, - 0x27, 0xca, 0xf0, 0x4f, 0x8e, 0x19, 0x26, 0x65, 0x07, 0xff, 0x34, 0xee, 0x42, 0xcb, 0xf1, 0x3c, - 0xcc, 0xa7, 0x3b, 0xe3, 0x03, 0xec, 0x25, 0x09, 0x24, 0x8f, 0x35, 0xff, 0x5a, 0x80, 0xf6, 0x1e, - 0x09, 0x27, 0x1f, 0xe2, 0x31, 0xca, 0x64, 0x37, 0xa1, 0xa4, 0xaa, 0x3a, 0xf8, 0x37, 0xaf, 0xa4, - 0xcf, 0xf0, 0x18, 0xc9, 0xb0, 0x97, 0xbb, 0xae, 0xca, 0x11, 0x22, 0xe4, 0xf5, 0x60, 0x72, 0x25, - 0xd8, 0x94, 0x83, 0xc7, 0xc4, 0x13, 0x3d, 0x83, 0x87, 0xa9, 0x9d, 0x5c, 0x00, 0x36, 0xad, 0x8a, - 0x87, 0xa9, 0x18, 0x52, 0x86, 0x2c, 0x89, 0x0b, 0xde, 0xac, 0x21, 0xcb, 0x12, 0xc3, 0x0d, 0x59, - 0x87, 0x65, 0x72, 0x76, 0x16, 0x21, 0x26, 0xd6, 0xaa, 0x64, 0x29, 0x28, 0x49, 0xc1, 0xd5, 0x4c, - 0x0a, 0x5e, 0x03, 0xe3, 0x00, 0xb1, 0x47, 0x8f, 0x8e, 0xf7, 0x2f, 0x50, 0xc0, 0xf4, 0xc9, 0xf5, - 0x3a, 0x54, 0x35, 0xea, 0xbf, 0xb9, 0x3a, 0x7d, 0x15, 0x5a, 0x3b, 0x9e, 0x77, 0xf2, 0xd8, 0x09, - 0xb5, 0x3f, 0xba, 0x50, 0xe9, 0xef, 0x1d, 0xf5, 0xa5, 0x4b, 0x4a, 0xdc, 0x00, 0x05, 0xf2, 0x93, - 0xf2, 0x00, 0xb1, 0x63, 0xc4, 0x28, 0x76, 0x93, 0x93, 0xf2, 0x0e, 0x54, 0x14, 0x86, 0xcf, 0xf4, - 0xe5, 0xa7, 0x3e, 0x02, 0x14, 0x68, 0xfe, 0x00, 0x8c, 0x1f, 0xf1, 0x9a, 0x0f, 0xc9, 0x82, 0x5f, - 0x49, 0x7a, 0x15, 0x3a, 0x17, 0x02, 0x6b, 0xcb, 0x62, 0x28, 0xb3, 0x0c, 0x6d, 0x39, 0x20, 0xf2, - 0x83, 0x90, 0x7d, 0x0a, 0xab, 0xb2, 0x44, 0x95, 0x7c, 0xae, 0xc1, 0x82, 0xfb, 0x30, 0x59, 0xcf, - 0xb2, 0x25, 0xbe, 0xef, 0xfd, 0xb9, 0xa3, 0x8e, 0x31, 0x75, 0x23, 0x62, 0x1c, 0x40, 0x7b, 0xea, - 0xf9, 0xca, 0x50, 0x57, 0x64, 0xf3, 0x5f, 0xb5, 0x7a, 0xeb, 0x5b, 0xf2, 0x39, 0x6c, 0x4b, 0x3f, - 0x87, 0x6d, 0xed, 0xfb, 0x21, 0x9b, 0x18, 0xfb, 0xd0, 0xca, 0x3f, 0xf4, 0x18, 0xcf, 0xea, 0x8a, - 0x72, 0xce, 0xf3, 0xcf, 0x42, 0x36, 0x07, 0xd0, 0x9e, 0x7a, 0xf3, 0xd1, 0xfa, 0xcc, 0x7f, 0x0a, - 0x5a, 0xc8, 0xe8, 0x01, 0xd4, 0x33, 0x8f, 0x3c, 0x46, 0x57, 0x32, 0x99, 0x7d, 0xf7, 0x59, 0xc8, - 0x60, 0x0f, 0x9a, 0xb9, 0x77, 0x17, 0xa3, 0xa7, 0xec, 0x99, 0xf3, 0x18, 0xb3, 0x90, 0xc9, 0x2e, - 0xd4, 0x33, 0xcf, 0x1f, 0x5a, 0x8b, 0xd9, 0x37, 0x96, 0xde, 0xad, 0x39, 0x23, 0xea, 0xb4, 0x3c, - 0x80, 0xf6, 0xd4, 0x9b, 0x88, 0x76, 0xc9, 0xfc, 0xa7, 0x92, 0x85, 0xca, 0x7c, 0x2c, 0x96, 0x28, - 0xd3, 0xf2, 0x66, 0x96, 0x68, 0xf6, 0x05, 0xa4, 0xf7, 0xdc, 0xfc, 0x41, 0xa5, 0xd5, 0x3e, 0xb4, - 0xf2, 0x8f, 0x1f, 0x9a, 0xd9, 0xdc, 0x27, 0x91, 0xab, 0xd7, 0x3b, 0xf7, 0x0e, 0x92, 0xae, 0xf7, - 0xbc, 0xe7, 0x91, 0x85, 0x8c, 0x76, 0x00, 0x54, 0x83, 0xeb, 0xe1, 0x20, 0x71, 0xf4, 0x4c, 0x63, - 0x9d, 0x38, 0x7a, 0x4e, 0x33, 0xfc, 0x00, 0x40, 0xf6, 0xa5, 0x1e, 0x89, 0x99, 0x71, 0x53, 0xab, - 0x31, 0xd5, 0x0c, 0xf7, 0xba, 0xb3, 0x03, 0x33, 0x0c, 0x10, 0xa5, 0xd7, 0x61, 0xf0, 0x01, 0x40, - 0xda, 0xef, 0x6a, 0x06, 0x33, 0x1d, 0xf0, 0x15, 0x3e, 0x68, 0x64, 0xbb, 0x5b, 0x43, 0xd9, 0x3a, - 0xa7, 0xe3, 0xbd, 0x82, 0x45, 0x7b, 0xaa, 0x7b, 0xc9, 0x6f, 0xb6, 0xe9, 0xa6, 0xa6, 0x37, 0xd3, - 0xc1, 0x18, 0xf7, 0xa1, 0x91, 0x6d, 0x5b, 0xb4, 0x16, 0x73, 0x5a, 0x99, 0x5e, 0xae, 0x75, 0x31, - 0x1e, 0x40, 0x2b, 0xdf, 0xb2, 0xe8, 0x2d, 0x35, 0xb7, 0x91, 0xe9, 0xa9, 0x0b, 0xb9, 0x0c, 0xf9, - 0x5b, 0x00, 0x69, 0x6b, 0xa3, 0xdd, 0x37, 0xd3, 0xec, 0x4c, 0x49, 0x3d, 0x80, 0xf6, 0x54, 0xcb, - 0xa2, 0x2d, 0x9e, 0xdf, 0xc9, 0x2c, 0x74, 0xdd, 0xdb, 0x00, 0xe9, 0x71, 0xa1, 0xa5, 0xcf, 0x1c, - 0x20, 0xbd, 0xa6, 0xbe, 0xac, 0x94, 0x74, 0x7b, 0xd0, 0xcc, 0xf5, 0xf3, 0x3a, 0xcd, 0xcc, 0x6b, - 0xf2, 0xaf, 0x4a, 0xbe, 0xf9, 0xe6, 0x57, 0x7b, 0x6e, 0x6e, 0x4b, 0x7c, 0xd5, 0xfe, 0xc9, 0x76, - 0x5c, 0x7a, 0xe5, 0xe6, 0x74, 0x61, 0xdf, 0x12, 0xcf, 0xd9, 0xae, 0x2a, 0x13, 0xcf, 0x73, 0x9a, - 0xad, 0x85, 0x8c, 0x0e, 0xa1, 0x7d, 0xa0, 0x0b, 0x66, 0x55, 0xcc, 0x2b, 0x75, 0xe6, 0x34, 0x2f, - 0xbd, 0xde, 0xbc, 0x21, 0x15, 0x54, 0x1f, 0x43, 0x67, 0xa6, 0x90, 0x37, 0x36, 0x92, 0x2b, 0xe3, - 0xb9, 0x15, 0xfe, 0x42, 0xb5, 0x8e, 0x60, 0x65, 0xba, 0x8e, 0x37, 0x9e, 0x57, 0x89, 0x72, 0x7e, - 0x7d, 0xbf, 0x90, 0xd5, 0xbb, 0x50, 0xd5, 0xb5, 0x99, 0xa1, 0xae, 0xe6, 0xa7, 0x6a, 0xb5, 0x85, - 0x53, 0xef, 0x43, 0x3d, 0x53, 0x0a, 0xe9, 0x6c, 0x37, 0x5b, 0x1d, 0xf5, 0xd4, 0x4d, 0x7a, 0x42, - 0x79, 0x1f, 0x2a, 0xaa, 0xfc, 0x31, 0xd6, 0x92, 0x4d, 0x9e, 0xa9, 0x86, 0xae, 0xda, 0x61, 0x07, - 0x88, 0x65, 0x8a, 0x1a, 0x2d, 0x74, 0xb6, 0xce, 0xd1, 0x29, 0x36, 0x37, 0xa2, 0xd6, 0x62, 0x07, - 0x1a, 0xd9, 0xb2, 0x46, 0x2f, 0xe9, 0x9c, 0x52, 0x67, 0x91, 0x26, 0xbb, 0x97, 0x5f, 0x7f, 0xb3, - 0xf1, 0xcc, 0xdf, 0xbf, 0xd9, 0x78, 0xe6, 0x57, 0x4f, 0x36, 0x0a, 0x5f, 0x3f, 0xd9, 0x28, 0xfc, - 0xed, 0xc9, 0x46, 0xe1, 0x5f, 0x4f, 0x36, 0x0a, 0x3f, 0xf9, 0xf9, 0xff, 0xf8, 0x1f, 0x21, 0x1a, - 0x07, 0x0c, 0xfb, 0x68, 0xfb, 0x02, 0x53, 0x96, 0x19, 0x0a, 0xcf, 0x87, 0xf2, 0x8f, 0x42, 0x99, - 0xff, 0x0f, 0x71, 0x2d, 0x07, 0xcb, 0x02, 0x7e, 0xeb, 0x3f, 0x01, 0x00, 0x00, 0xff, 0xff, 0x6c, - 0xf4, 0x1d, 0x49, 0x8c, 0x24, 0x00, 0x00, + // 3207 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x3a, 0x4b, 0x73, 0x1b, 0xc7, + 0xd1, 0xc6, 0x83, 0x04, 0xd0, 0x78, 0x11, 0x4b, 0x8a, 0x82, 0x60, 0x9b, 0x96, 0x57, 0xb6, 0x4c, + 0xd9, 0x9f, 0x48, 0x5b, 0x76, 0x59, 0x7e, 0x94, 0x3f, 0x7d, 0x24, 0x45, 0x93, 0xb4, 0x4d, 0x0b, + 0xdf, 0x42, 0x8c, 0x53, 0x49, 0x25, 0x5b, 0xcb, 0xdd, 0x21, 0x38, 0x26, 0x76, 0x67, 0x3d, 0x3b, + 0x0b, 0x91, 0x4e, 0x55, 0x2a, 0xa7, 0xe4, 0x96, 0x63, 0x6e, 0xf9, 0x03, 0xa9, 0xfc, 0x83, 0x5c, + 0x73, 0x50, 0xe5, 0x94, 0x63, 0x2e, 0x49, 0xc5, 0xfa, 0x09, 0xf9, 0x05, 0xa9, 0x79, 0xed, 0x03, + 0x0f, 0xda, 0x61, 0xa9, 0x2a, 0x17, 0xd4, 0x76, 0x4f, 0x4f, 0xbf, 0xa6, 0xa7, 0xa7, 0x7b, 0x06, + 0xd0, 0x1f, 0x62, 0x76, 0x1a, 0x1f, 0x6f, 0xb8, 0xc4, 0xdf, 0x3c, 0x73, 0x98, 0x73, 0xd7, 0x25, + 0x01, 0x73, 0x70, 0x80, 0x68, 0x34, 0x05, 0x47, 0xd4, 0xdd, 0x1c, 0xe1, 0xe3, 0x68, 0x33, 0xa4, + 0x84, 0x11, 0x97, 0x8c, 0xd4, 0x57, 0xb4, 0xe9, 0x0c, 0x51, 0xc0, 0x36, 0x04, 0x60, 0x94, 0x87, + 0x34, 0x74, 0x7b, 0x35, 0xe2, 0x62, 0x89, 0xe8, 0xd5, 0xdc, 0x48, 0x7f, 0xd6, 0xd9, 0x45, 0x88, + 0x22, 0x05, 0xbc, 0x38, 0x24, 0x64, 0x38, 0x42, 0x92, 0xc7, 0x71, 0x7c, 0xb2, 0x89, 0xfc, 0x90, + 0x5d, 0xc8, 0x41, 0xf3, 0xf7, 0x45, 0x58, 0xdd, 0xa1, 0xc8, 0x61, 0x68, 0x47, 0x2b, 0x60, 0xa1, + 0x6f, 0x62, 0x14, 0x31, 0xe3, 0x55, 0x68, 0x24, 0x4a, 0xd9, 0xd8, 0xeb, 0x16, 0x6e, 0x16, 0xd6, + 0x6b, 0x56, 0x3d, 0xc1, 0x1d, 0x78, 0xc6, 0x75, 0xa8, 0xa0, 0x73, 0xe4, 0xf2, 0xd1, 0xa2, 0x18, + 0x5d, 0xe4, 0xe0, 0x81, 0x67, 0xbc, 0x03, 0xf5, 0x88, 0x51, 0x1c, 0x0c, 0xed, 0x38, 0x42, 0xb4, + 0x5b, 0xba, 0x59, 0x58, 0xaf, 0xdf, 0x5b, 0xda, 0xe0, 0x2a, 0x6f, 0x0c, 0xc4, 0xc0, 0x51, 0x84, + 0xa8, 0x05, 0x51, 0xf2, 0x6d, 0xdc, 0x86, 0x8a, 0x87, 0xc6, 0xd8, 0x45, 0x51, 0xb7, 0x7c, 0xb3, + 0xb4, 0x5e, 0xbf, 0xd7, 0x90, 0xe4, 0x0f, 0x05, 0xd2, 0xd2, 0x83, 0xc6, 0x1d, 0xa8, 0x46, 0x8c, + 0x50, 0x67, 0x88, 0xa2, 0xee, 0x82, 0x20, 0x6c, 0x6a, 0xbe, 0x02, 0x6b, 0x25, 0xc3, 0xc6, 0x4b, + 0x50, 0x7a, 0xb4, 0x73, 0xd0, 0x5d, 0x14, 0xd2, 0x41, 0x51, 0x85, 0xc8, 0xb5, 0x38, 0xda, 0xb8, + 0x05, 0xcd, 0xc8, 0x09, 0xbc, 0x63, 0x72, 0x6e, 0x87, 0xd8, 0x0b, 0xa2, 0x6e, 0xe5, 0x66, 0x61, + 0xbd, 0x6a, 0x35, 0x14, 0xb2, 0xcf, 0x71, 0xe6, 0x47, 0x70, 0x6d, 0xc0, 0x1c, 0xca, 0xae, 0xe0, + 0x1d, 0xf3, 0x08, 0x56, 0x2d, 0xe4, 0x93, 0xf1, 0x95, 0x5c, 0xdb, 0x85, 0x0a, 0xc3, 0x3e, 0x22, + 0x31, 0x13, 0xae, 0x6d, 0x5a, 0x1a, 0x34, 0xff, 0x58, 0x00, 0x63, 0xf7, 0x1c, 0xb9, 0x7d, 0x4a, + 0x5c, 0x14, 0x45, 0xff, 0xa5, 0xe5, 0x7a, 0x03, 0x2a, 0xa1, 0x54, 0xa0, 0x5b, 0x16, 0xe4, 0x6a, + 0x15, 0xb4, 0x56, 0x7a, 0xd4, 0xfc, 0x1a, 0x56, 0x06, 0x78, 0x18, 0x38, 0xa3, 0xe7, 0xa8, 0xef, + 0x2a, 0x2c, 0x46, 0x82, 0xa7, 0x50, 0xb5, 0x69, 0x29, 0xc8, 0xec, 0x83, 0xf1, 0x95, 0x83, 0xd9, + 0xf3, 0x93, 0x64, 0xde, 0x85, 0xe5, 0x1c, 0xc7, 0x28, 0x24, 0x41, 0x84, 0x84, 0x02, 0xcc, 0x61, + 0x71, 0x24, 0x98, 0x2d, 0x58, 0x0a, 0x32, 0x09, 0xac, 0x1e, 0x85, 0xde, 0x15, 0x77, 0xd3, 0x3d, + 0xa8, 0x51, 0x14, 0x91, 0x98, 0xf2, 0x3d, 0x50, 0x14, 0x4e, 0x5d, 0x91, 0x4e, 0xfd, 0x02, 0x07, + 0xf1, 0xb9, 0xa5, 0xc7, 0xac, 0x94, 0x4c, 0xc5, 0x27, 0x8b, 0xae, 0x12, 0x9f, 0x1f, 0xc1, 0xb5, + 0xbe, 0x13, 0x47, 0x57, 0xd1, 0xd5, 0xfc, 0x98, 0xc7, 0x76, 0x14, 0xfb, 0x57, 0x9a, 0xfc, 0x87, + 0x02, 0x54, 0x77, 0xc2, 0xf8, 0x28, 0x72, 0x86, 0xc8, 0x78, 0x05, 0xea, 0x8c, 0x30, 0x67, 0x64, + 0xc7, 0x1c, 0x14, 0xe4, 0x65, 0x0b, 0x04, 0x4a, 0x12, 0xbc, 0x0a, 0x8d, 0x10, 0x51, 0x37, 0x8c, + 0x15, 0x45, 0xf1, 0x66, 0x69, 0xbd, 0x6c, 0xd5, 0x25, 0x4e, 0x92, 0x6c, 0xc0, 0xb2, 0x18, 0xb3, + 0x71, 0x60, 0x9f, 0x21, 0x1a, 0xa0, 0x91, 0x4f, 0x3c, 0x24, 0x82, 0xa3, 0x6c, 0x75, 0xc4, 0xd0, + 0x41, 0xf0, 0x79, 0x32, 0x60, 0xbc, 0x09, 0x9d, 0x84, 0x9e, 0x47, 0xbc, 0xa0, 0x2e, 0x0b, 0xea, + 0xb6, 0xa2, 0x3e, 0x52, 0x68, 0xf3, 0x97, 0xd0, 0x7a, 0x7c, 0x4a, 0x09, 0x63, 0x23, 0x1c, 0x0c, + 0x1f, 0x3a, 0xcc, 0xe1, 0x5b, 0x33, 0x44, 0x14, 0x13, 0x2f, 0x52, 0xda, 0x6a, 0xd0, 0x78, 0x0b, + 0x3a, 0x4c, 0xd2, 0x22, 0xcf, 0xd6, 0x34, 0x45, 0x41, 0xb3, 0x94, 0x0c, 0xf4, 0x15, 0xf1, 0xeb, + 0xd0, 0x4a, 0x89, 0xf9, 0xe6, 0x56, 0xfa, 0x36, 0x13, 0xec, 0x63, 0xec, 0x23, 0x73, 0x2c, 0x7c, + 0x25, 0x16, 0xd9, 0x78, 0x0b, 0x6a, 0xa9, 0x1f, 0x0a, 0x22, 0x42, 0x5a, 0x32, 0x42, 0xb4, 0x3b, + 0xad, 0x6a, 0xe2, 0x94, 0x4f, 0xa0, 0xcd, 0x12, 0xc5, 0x6d, 0xcf, 0x61, 0x4e, 0x3e, 0xa8, 0xf2, + 0x56, 0x59, 0x2d, 0x96, 0x83, 0xcd, 0x8f, 0xa1, 0xd6, 0xc7, 0x5e, 0x24, 0x05, 0x77, 0xa1, 0xe2, + 0xc6, 0x94, 0xa2, 0x80, 0x69, 0x93, 0x15, 0x68, 0xac, 0xc0, 0xc2, 0x08, 0xfb, 0x98, 0x29, 0x33, + 0x25, 0x60, 0x12, 0x80, 0x43, 0xe4, 0x13, 0x7a, 0x21, 0x1c, 0xb6, 0x02, 0x0b, 0xd9, 0xc5, 0x95, + 0x80, 0xf1, 0x22, 0xd4, 0x7c, 0xe7, 0x3c, 0x59, 0x54, 0x3e, 0x52, 0xf5, 0x9d, 0x73, 0xa9, 0x7c, + 0x17, 0x2a, 0x27, 0x0e, 0x1e, 0xb9, 0x01, 0x53, 0x5e, 0xd1, 0x60, 0x2a, 0xb0, 0x9c, 0x15, 0xf8, + 0xe7, 0x22, 0xd4, 0xa5, 0x44, 0xa9, 0xf0, 0x0a, 0x2c, 0xb8, 0x8e, 0x7b, 0x9a, 0x88, 0x14, 0x80, + 0x71, 0x5b, 0x2b, 0x52, 0xcc, 0x66, 0xb8, 0x54, 0x53, 0xad, 0xda, 0x26, 0x40, 0xf4, 0xc4, 0x09, + 0x95, 0x6e, 0xa5, 0x39, 0xc4, 0x35, 0x4e, 0x23, 0xd5, 0x7d, 0x17, 0x1a, 0x32, 0xee, 0xd4, 0x94, + 0xf2, 0x9c, 0x29, 0x75, 0x49, 0x25, 0x27, 0xdd, 0x82, 0x66, 0x1c, 0x21, 0xfb, 0x14, 0x23, 0xea, + 0x50, 0xf7, 0xf4, 0xa2, 0xbb, 0x20, 0x0f, 0xa0, 0x38, 0x42, 0xfb, 0x1a, 0x67, 0xdc, 0x83, 0x05, + 0x9e, 0x5b, 0xa2, 0xee, 0xa2, 0x38, 0xeb, 0x5e, 0xca, 0xb2, 0x14, 0xa6, 0x6e, 0x88, 0xdf, 0xdd, + 0x80, 0xd1, 0x0b, 0x4b, 0x92, 0xf6, 0x3e, 0x00, 0x48, 0x91, 0xc6, 0x12, 0x94, 0xce, 0xd0, 0x85, + 0xda, 0x87, 0xfc, 0x93, 0x3b, 0x67, 0xec, 0x8c, 0x62, 0xed, 0x75, 0x09, 0x7c, 0x54, 0xfc, 0xa0, + 0x60, 0xba, 0xd0, 0xde, 0x1e, 0x9d, 0x61, 0x92, 0x99, 0xbe, 0x02, 0x0b, 0xbe, 0xf3, 0x35, 0xa1, + 0xda, 0x93, 0x02, 0x10, 0x58, 0x1c, 0x10, 0xaa, 0x59, 0x08, 0xc0, 0x68, 0x41, 0x91, 0x84, 0xc2, + 0x5f, 0x35, 0xab, 0x48, 0xc2, 0x54, 0x50, 0x39, 0x23, 0xc8, 0xfc, 0x47, 0x19, 0x20, 0x95, 0x62, + 0x58, 0xd0, 0xc3, 0xc4, 0x8e, 0x10, 0xe5, 0xe7, 0xbb, 0x7d, 0x7c, 0xc1, 0x50, 0x64, 0x53, 0xe4, + 0xc6, 0x34, 0xc2, 0x63, 0xbe, 0x7e, 0xdc, 0xec, 0x6b, 0xd2, 0xec, 0x09, 0xdd, 0xac, 0xeb, 0x98, + 0x0c, 0xe4, 0xbc, 0x6d, 0x3e, 0xcd, 0xd2, 0xb3, 0x8c, 0x03, 0xb8, 0x96, 0xf2, 0xf4, 0x32, 0xec, + 0x8a, 0x97, 0xb1, 0x5b, 0x4e, 0xd8, 0x79, 0x29, 0xab, 0x5d, 0x58, 0xc6, 0xc4, 0xfe, 0x26, 0x46, + 0x71, 0x8e, 0x51, 0xe9, 0x32, 0x46, 0x1d, 0x4c, 0xfe, 0x5f, 0x4c, 0x48, 0xd9, 0xf4, 0xe1, 0x46, + 0xc6, 0x4a, 0xbe, 0xdd, 0x33, 0xcc, 0xca, 0x97, 0x31, 0x5b, 0x4d, 0xb4, 0xe2, 0xf9, 0x20, 0xe5, + 0xf8, 0x19, 0xac, 0x62, 0x62, 0x3f, 0x71, 0x30, 0x9b, 0x64, 0xb7, 0xf0, 0x3d, 0x46, 0xf2, 0x13, + 0x2d, 0xcf, 0x4b, 0x1a, 0xe9, 0x23, 0x3a, 0xcc, 0x19, 0xb9, 0xf8, 0x3d, 0x46, 0x1e, 0x8a, 0x09, + 0x29, 0x9b, 0x2d, 0xe8, 0x60, 0x32, 0xa9, 0x4d, 0xe5, 0x32, 0x26, 0x6d, 0x4c, 0xf2, 0x9a, 0x6c, + 0x43, 0x27, 0x42, 0x2e, 0x23, 0x34, 0x1b, 0x04, 0xd5, 0xcb, 0x58, 0x2c, 0x29, 0xfa, 0x84, 0x87, + 0xf9, 0x53, 0x68, 0xec, 0xc7, 0x43, 0xc4, 0x46, 0xc7, 0x49, 0x32, 0x78, 0x6e, 0xf9, 0xc7, 0xfc, + 0x57, 0x11, 0xea, 0x3b, 0x43, 0x4a, 0xe2, 0x30, 0x97, 0x93, 0xe5, 0x26, 0x9d, 0xcc, 0xc9, 0x82, + 0x44, 0xe4, 0x64, 0x49, 0xfc, 0x1e, 0x34, 0x7c, 0xb1, 0x75, 0x15, 0xbd, 0xcc, 0x43, 0x9d, 0xa9, + 0x4d, 0x6d, 0xd5, 0xfd, 0x4c, 0x32, 0xdb, 0x00, 0x08, 0xb1, 0x17, 0xa9, 0x39, 0x32, 0x1d, 0xb5, + 0x55, 0xb9, 0xa5, 0x53, 0xb4, 0x55, 0x0b, 0x93, 0x6c, 0xfd, 0x0e, 0xd4, 0x8f, 0xb9, 0x93, 0xd4, + 0x84, 0x5c, 0x32, 0x4a, 0xbd, 0x67, 0xc1, 0x71, 0xba, 0x09, 0xf7, 0xa1, 0x79, 0x2a, 0x5d, 0xa6, + 0x26, 0xc9, 0x18, 0xba, 0xa5, 0x2c, 0x49, 0xed, 0xdd, 0xc8, 0x7a, 0x56, 0x2e, 0x40, 0xe3, 0x34, + 0x83, 0xea, 0x0d, 0xa0, 0x33, 0x45, 0x32, 0x23, 0x07, 0xad, 0x67, 0x73, 0x50, 0xfd, 0x9e, 0x21, + 0x05, 0x65, 0x67, 0x66, 0xf3, 0xd2, 0x6f, 0x8b, 0xd0, 0xf8, 0x12, 0xb1, 0x27, 0x84, 0x9e, 0x49, + 0x7d, 0x0d, 0x28, 0x07, 0x8e, 0x8f, 0x14, 0x47, 0xf1, 0x6d, 0xdc, 0x80, 0x2a, 0x3d, 0x97, 0x09, + 0x44, 0xad, 0x67, 0x85, 0x9e, 0x8b, 0xc4, 0x60, 0xbc, 0x0c, 0x40, 0xcf, 0xed, 0xd0, 0x71, 0xcf, + 0x90, 0xf2, 0x60, 0xd9, 0xaa, 0xd1, 0xf3, 0xbe, 0x44, 0xf0, 0x50, 0xa0, 0xe7, 0x36, 0xa2, 0x94, + 0xd0, 0x48, 0xe5, 0xaa, 0x2a, 0x3d, 0xdf, 0x15, 0xb0, 0x9a, 0xeb, 0x51, 0x12, 0x86, 0xc8, 0x13, + 0x39, 0x5a, 0xcc, 0x7d, 0x28, 0x11, 0x5c, 0x2a, 0xd3, 0x52, 0x17, 0xa5, 0x54, 0x96, 0x4a, 0x65, + 0xa9, 0xd4, 0x8a, 0x9c, 0xc9, 0xb2, 0x52, 0x59, 0x22, 0xb5, 0x2a, 0xa5, 0xb2, 0x8c, 0x54, 0x96, + 0x4a, 0xad, 0xe9, 0xb9, 0x4a, 0xaa, 0xf9, 0x9b, 0x02, 0xac, 0x4e, 0x16, 0x7e, 0xaa, 0x36, 0x7d, + 0x0f, 0x1a, 0xae, 0x58, 0xaf, 0x5c, 0x4c, 0x76, 0xa6, 0x56, 0xd2, 0xaa, 0xbb, 0x99, 0x30, 0xbe, + 0x0f, 0xcd, 0x40, 0x3a, 0x38, 0x09, 0xcd, 0x52, 0xba, 0x2e, 0x59, 0xdf, 0x5b, 0x8d, 0x20, 0x03, + 0x99, 0x1e, 0x18, 0x5f, 0x51, 0xcc, 0xd0, 0x80, 0x51, 0xe4, 0xf8, 0xcf, 0xa3, 0xba, 0x37, 0xa0, + 0x2c, 0xaa, 0x15, 0xbe, 0x4c, 0x0d, 0x4b, 0x7c, 0x9b, 0x6f, 0xc0, 0x72, 0x4e, 0x8a, 0xb2, 0x75, + 0x09, 0x4a, 0x23, 0x14, 0x08, 0xee, 0x4d, 0x8b, 0x7f, 0x9a, 0x0e, 0x74, 0x2c, 0xe4, 0x78, 0xcf, + 0x4f, 0x1b, 0x25, 0xa2, 0x94, 0x8a, 0x58, 0x07, 0x23, 0x2b, 0x42, 0xa9, 0xa2, 0xb5, 0x2e, 0x64, + 0xb4, 0x7e, 0x04, 0x9d, 0x9d, 0x11, 0x89, 0xd0, 0x80, 0x79, 0x38, 0x78, 0x1e, 0xed, 0xc8, 0x2f, + 0x60, 0xf9, 0x31, 0xbb, 0xf8, 0x8a, 0x33, 0x8b, 0xf0, 0xb7, 0xe8, 0x39, 0xd9, 0x47, 0xc9, 0x13, + 0x6d, 0x1f, 0x25, 0x4f, 0x78, 0x73, 0xe3, 0x92, 0x51, 0xec, 0x07, 0x62, 0x2b, 0x34, 0x2d, 0x05, + 0x99, 0xdb, 0xd0, 0x90, 0x35, 0xf4, 0x21, 0xf1, 0xe2, 0x11, 0x9a, 0xb9, 0x07, 0xd7, 0x00, 0x42, + 0x87, 0x3a, 0x3e, 0x62, 0x88, 0xca, 0x18, 0xaa, 0x59, 0x19, 0x8c, 0xf9, 0xbb, 0x22, 0xac, 0xc8, + 0xfb, 0x86, 0x81, 0x6c, 0xb3, 0xb5, 0x09, 0x3d, 0xa8, 0x9e, 0x92, 0x88, 0x65, 0x18, 0x26, 0x30, + 0x57, 0x91, 0xf7, 0xe7, 0x92, 0x1b, 0xff, 0xcc, 0x5d, 0x02, 0x94, 0x2e, 0xbf, 0x04, 0x98, 0x6a, + 0xf3, 0xcb, 0xd3, 0x6d, 0x3e, 0xdf, 0x6d, 0x9a, 0x08, 0xcb, 0x3d, 0x5e, 0xb3, 0x6a, 0x0a, 0x73, + 0xe0, 0x19, 0xb7, 0xa1, 0x3d, 0xe4, 0x5a, 0xda, 0xa7, 0x84, 0x9c, 0xd9, 0xa1, 0xc3, 0x4e, 0xc5, + 0x56, 0xaf, 0x59, 0x4d, 0x81, 0xde, 0x27, 0xe4, 0xac, 0xef, 0xb0, 0x53, 0xe3, 0x43, 0x68, 0xa9, + 0x32, 0xd0, 0x17, 0x2e, 0x8a, 0xd4, 0xe1, 0xa7, 0x76, 0x51, 0xd6, 0x7b, 0x56, 0xf3, 0x2c, 0x03, + 0x45, 0xe6, 0x75, 0xb8, 0xf6, 0x10, 0x45, 0x8c, 0x92, 0x8b, 0xbc, 0x63, 0xcc, 0xff, 0x05, 0x38, + 0x08, 0x18, 0xa2, 0x27, 0x8e, 0x8b, 0x22, 0xe3, 0xed, 0x2c, 0xa4, 0x8a, 0xa3, 0xa5, 0x0d, 0x79, + 0xdd, 0x93, 0x0c, 0x58, 0x19, 0x1a, 0x73, 0x03, 0x16, 0x2d, 0x12, 0xf3, 0x74, 0xf4, 0x9a, 0xfe, + 0x52, 0xf3, 0x1a, 0x6a, 0x9e, 0x40, 0x5a, 0x6a, 0xcc, 0xdc, 0xd7, 0x2d, 0x6c, 0xca, 0x4e, 0x2d, + 0xd1, 0x06, 0xd4, 0xb0, 0xc6, 0xa9, 0xac, 0x32, 0x2d, 0x3a, 0x25, 0x31, 0x3f, 0x86, 0x65, 0xc9, + 0x49, 0x72, 0xd6, 0x6c, 0x5e, 0x83, 0x45, 0xaa, 0xd5, 0x28, 0xa4, 0xf7, 0x3c, 0x8a, 0x48, 0x8d, + 0x71, 0x7f, 0x7c, 0x81, 0x23, 0x96, 0x1a, 0xa2, 0xfd, 0xb1, 0x0c, 0x1d, 0x3e, 0x90, 0xe3, 0x69, + 0x7e, 0x0a, 0x8d, 0x2d, 0xab, 0xff, 0x25, 0xc2, 0xc3, 0xd3, 0x63, 0x9e, 0x3d, 0xdf, 0xcf, 0xc3, + 0xca, 0x60, 0x43, 0x69, 0x9b, 0x19, 0xb2, 0x72, 0x74, 0xe6, 0x67, 0xb0, 0xba, 0xe5, 0x79, 0x59, + 0x94, 0xd6, 0xfa, 0x6d, 0xa8, 0x05, 0x19, 0x76, 0x99, 0x33, 0x2b, 0x47, 0x9d, 0x12, 0x99, 0x77, + 0xc1, 0xd8, 0x43, 0xec, 0xa0, 0xff, 0xd8, 0x39, 0x1e, 0xa5, 0xd6, 0x5f, 0x87, 0x0a, 0x8e, 0x6c, + 0x1c, 0x8e, 0xdf, 0x17, 0x5c, 0xaa, 0xd6, 0x22, 0x8e, 0x0e, 0xc2, 0xf1, 0xfb, 0xe6, 0x1d, 0x58, + 0xce, 0x91, 0x5f, 0x92, 0x56, 0xb6, 0xc0, 0x18, 0xfc, 0x70, 0xce, 0x09, 0x8b, 0x62, 0x86, 0xc5, + 0x1d, 0x58, 0x1e, 0xfc, 0x40, 0x69, 0x3f, 0x83, 0xe5, 0x47, 0xc1, 0x08, 0x07, 0x68, 0xa7, 0x7f, + 0x74, 0x88, 0x92, 0x9c, 0x6a, 0x40, 0x99, 0xd7, 0x9e, 0x4a, 0x96, 0xf8, 0xe6, 0x2a, 0x04, 0xc7, + 0xb6, 0x1b, 0xc6, 0x91, 0xba, 0xb4, 0x5a, 0x0c, 0x8e, 0x77, 0xc2, 0x38, 0xe2, 0x87, 0x24, 0x2f, + 0x92, 0x48, 0x30, 0xba, 0x10, 0x99, 0xa6, 0x6a, 0x55, 0xdc, 0x30, 0x7e, 0x14, 0x8c, 0x2e, 0xcc, + 0xff, 0x11, 0x37, 0x09, 0x08, 0x79, 0x96, 0x13, 0x78, 0xc4, 0x7f, 0x88, 0xc6, 0x19, 0x09, 0x53, + 0x7a, 0x3f, 0x2d, 0x40, 0x63, 0x6b, 0x88, 0x02, 0xf6, 0x10, 0x31, 0x07, 0x8f, 0x44, 0x67, 0x3a, + 0x46, 0x34, 0xc2, 0x24, 0x50, 0x69, 0x43, 0x83, 0xc6, 0x2b, 0x50, 0xc7, 0x01, 0x66, 0xb6, 0xe7, + 0x20, 0x9f, 0x04, 0x82, 0x4b, 0xd5, 0x02, 0x8e, 0x7a, 0x28, 0x30, 0xc6, 0x1b, 0xd0, 0x96, 0x97, + 0x8a, 0xf6, 0xa9, 0x13, 0x78, 0x23, 0x9e, 0xb0, 0x4a, 0x22, 0xc5, 0xb4, 0x24, 0x7a, 0x5f, 0x61, + 0x8d, 0x3b, 0xb0, 0xa4, 0xd2, 0x49, 0x4a, 0x59, 0x16, 0x94, 0x6d, 0x85, 0xcf, 0x91, 0xc6, 0x61, + 0x48, 0x28, 0x8b, 0xec, 0x08, 0xb9, 0x2e, 0xf1, 0x43, 0xd5, 0xd6, 0xb5, 0x35, 0x7e, 0x20, 0xd1, + 0xe6, 0x10, 0x96, 0xf7, 0xb8, 0x9d, 0xca, 0x92, 0x74, 0x7b, 0xb4, 0x7c, 0xe4, 0xdb, 0xc7, 0x23, + 0xe2, 0x9e, 0xd9, 0x3c, 0xc9, 0x2b, 0x0f, 0xf3, 0xc2, 0x71, 0x9b, 0x23, 0x07, 0xf8, 0x5b, 0x71, + 0x83, 0xc1, 0xa9, 0x4e, 0x09, 0x0b, 0x47, 0xf1, 0xd0, 0x0e, 0x29, 0x39, 0x46, 0xca, 0xc4, 0xb6, + 0x8f, 0xfc, 0x7d, 0x89, 0xef, 0x73, 0xb4, 0xf9, 0xa7, 0x02, 0xac, 0xe4, 0x25, 0xa9, 0xd5, 0xde, + 0x84, 0x95, 0xbc, 0x28, 0x55, 0xc6, 0xc8, 0x32, 0xb9, 0x93, 0x15, 0x28, 0x0b, 0x9a, 0xfb, 0xd0, + 0x14, 0x57, 0xd0, 0xb6, 0x27, 0x39, 0xe5, 0x8b, 0xb7, 0xec, 0xba, 0x58, 0x0d, 0x27, 0xbb, 0x4a, + 0x1f, 0xc2, 0x0d, 0x65, 0xbe, 0x3d, 0xad, 0xb6, 0x0c, 0x88, 0x55, 0x45, 0x70, 0x38, 0xa1, 0xfd, + 0x17, 0xd0, 0x4d, 0x51, 0xdb, 0x17, 0x02, 0x99, 0x6e, 0xca, 0xe5, 0x09, 0x63, 0xb7, 0x3c, 0x8f, + 0x8a, 0xdd, 0x5e, 0xb6, 0x66, 0x0d, 0x99, 0x0f, 0xe0, 0xfa, 0x00, 0x31, 0xe9, 0x0d, 0x87, 0xa9, + 0x8e, 0x4a, 0x32, 0x5b, 0x82, 0xd2, 0x00, 0xb9, 0xc2, 0xf8, 0x92, 0xc5, 0x3f, 0x79, 0x00, 0x1e, + 0x45, 0xc8, 0x15, 0x56, 0x96, 0x2c, 0xf1, 0x6d, 0x86, 0x50, 0xf9, 0x74, 0xb0, 0xc7, 0xeb, 0x26, + 0x1e, 0xd4, 0xb2, 0xce, 0x52, 0x67, 0x6a, 0xd3, 0xaa, 0x08, 0xf8, 0xc0, 0x33, 0x3e, 0x83, 0x65, + 0x39, 0xe4, 0x9e, 0x3a, 0xc1, 0x10, 0xd9, 0x21, 0x19, 0x61, 0x57, 0x86, 0x7e, 0xeb, 0x5e, 0x4f, + 0xa5, 0x21, 0xc5, 0x67, 0x47, 0x90, 0xf4, 0x05, 0x85, 0xd5, 0x19, 0x4e, 0xa2, 0xcc, 0xbf, 0x17, + 0xa0, 0xa2, 0x8e, 0x35, 0x7e, 0x34, 0x7b, 0x14, 0x8f, 0x11, 0x55, 0xc1, 0xae, 0x20, 0xe3, 0x75, + 0x68, 0xc9, 0x2f, 0x9b, 0x84, 0x0c, 0x93, 0xe4, 0xb0, 0x6c, 0x4a, 0xec, 0x23, 0x89, 0x14, 0xd7, + 0x96, 0xe2, 0xe2, 0x50, 0xf5, 0xe8, 0x0a, 0xe2, 0xf8, 0x93, 0x88, 0x2b, 0x25, 0x0e, 0xc7, 0x9a, + 0xa5, 0x20, 0xbe, 0xb9, 0x34, 0xbf, 0x05, 0xc1, 0x4f, 0x83, 0x7c, 0x73, 0xf9, 0x24, 0x0e, 0x98, + 0x1d, 0x12, 0x1c, 0x30, 0x75, 0x1a, 0x82, 0x40, 0xf5, 0x39, 0xc6, 0x58, 0x87, 0xea, 0x49, 0x64, + 0x0b, 0x6b, 0x44, 0xe5, 0x9b, 0x9c, 0xd0, 0xca, 0x6a, 0xab, 0x72, 0x12, 0x89, 0x0f, 0xf3, 0xd7, + 0x05, 0x58, 0x94, 0x97, 0xfc, 0x46, 0x0b, 0x8a, 0x49, 0xf5, 0x52, 0xc4, 0xa2, 0x12, 0x14, 0x5a, + 0xc9, 0x8a, 0x45, 0x7c, 0xf3, 0x1c, 0x33, 0xf6, 0xe5, 0x19, 0xac, 0x8c, 0x18, 0xfb, 0xe2, 0xf0, + 0x7d, 0x1d, 0x5a, 0x69, 0x11, 0x24, 0xc6, 0xa5, 0x31, 0xcd, 0x04, 0x2b, 0xc8, 0xe6, 0xda, 0x64, + 0xfe, 0x18, 0x20, 0xbd, 0xec, 0xe6, 0xe1, 0x10, 0x27, 0xca, 0xf0, 0x4f, 0x8e, 0x19, 0x26, 0xe5, + 0x13, 0xff, 0x34, 0x6e, 0x43, 0xcb, 0xf1, 0x3c, 0xcc, 0xa7, 0x3b, 0xa3, 0x3d, 0xec, 0x25, 0x09, + 0x24, 0x8f, 0x35, 0xff, 0x52, 0x80, 0xf6, 0x0e, 0x09, 0x2f, 0x3e, 0xc5, 0x23, 0x94, 0xc9, 0x6e, + 0x42, 0x49, 0x55, 0x3d, 0xf1, 0x6f, 0xde, 0x11, 0x9c, 0xe0, 0x11, 0x92, 0xdb, 0x5e, 0x46, 0x5d, + 0x95, 0x23, 0xc4, 0x96, 0xd7, 0x83, 0xc9, 0xd5, 0x66, 0x53, 0x0e, 0x1e, 0x12, 0x4f, 0xf4, 0x3e, + 0x1e, 0xa6, 0x76, 0x72, 0x91, 0xd9, 0xb4, 0x2a, 0x1e, 0xa6, 0x62, 0x48, 0x19, 0xb2, 0x20, 0x2e, + 0xaa, 0xb3, 0x86, 0x2c, 0x4a, 0x0c, 0x37, 0x64, 0x15, 0x16, 0xc9, 0xc9, 0x49, 0x84, 0x98, 0x58, + 0xab, 0x92, 0xa5, 0xa0, 0x24, 0x05, 0x57, 0x33, 0x29, 0x78, 0x45, 0x9c, 0x6b, 0x8f, 0x1e, 0x1d, + 0xee, 0x8e, 0x51, 0xc0, 0xf4, 0x09, 0x7c, 0x17, 0xaa, 0x1a, 0xf5, 0x43, 0xae, 0x80, 0xdf, 0x84, + 0xd6, 0x96, 0xe7, 0x0d, 0x9e, 0x38, 0xa1, 0xf6, 0x47, 0x17, 0x2a, 0xfd, 0x9d, 0x83, 0xbe, 0x74, + 0x49, 0x89, 0x1b, 0xa0, 0x40, 0x7e, 0xe2, 0xef, 0x21, 0x76, 0x88, 0x18, 0xc5, 0x6e, 0x72, 0xe2, + 0xdf, 0x82, 0x8a, 0xc2, 0xf0, 0x99, 0xbe, 0xfc, 0xd4, 0x47, 0x80, 0x02, 0xcd, 0xff, 0x03, 0xe3, + 0x47, 0xbc, 0x76, 0x45, 0xb2, 0x71, 0x51, 0x92, 0xde, 0x84, 0xce, 0x58, 0x60, 0x6d, 0x59, 0xd4, + 0x65, 0x96, 0xa1, 0x2d, 0x07, 0x44, 0x7e, 0x10, 0xb2, 0x8f, 0x60, 0x59, 0x96, 0xda, 0x92, 0xcf, + 0x15, 0x58, 0x70, 0x1f, 0x26, 0xeb, 0x59, 0xb6, 0xc4, 0xf7, 0xbd, 0xa7, 0x86, 0x3a, 0xc6, 0xd4, + 0xcd, 0x8e, 0xb1, 0x07, 0xed, 0x89, 0x67, 0x38, 0x43, 0x5d, 0xf5, 0xcd, 0x7e, 0x9d, 0xeb, 0xad, + 0x6e, 0xc8, 0x67, 0xbd, 0x0d, 0xfd, 0xac, 0xb7, 0xb1, 0xeb, 0x87, 0xec, 0xc2, 0xd8, 0x85, 0x56, + 0xfe, 0xc1, 0xca, 0x78, 0x51, 0x57, 0xc6, 0x33, 0x9e, 0xb1, 0xe6, 0xb2, 0xd9, 0x83, 0xf6, 0xc4, + 0xdb, 0x95, 0xd6, 0x67, 0xf6, 0x93, 0xd6, 0x5c, 0x46, 0x0f, 0xa0, 0x9e, 0x79, 0xac, 0x32, 0xba, + 0x92, 0xc9, 0xf4, 0xfb, 0xd5, 0x5c, 0x06, 0x3b, 0xd0, 0xcc, 0xbd, 0x1f, 0x19, 0x3d, 0x65, 0xcf, + 0x8c, 0x47, 0xa5, 0xb9, 0x4c, 0xb6, 0xa1, 0x9e, 0x79, 0xc6, 0xd1, 0x5a, 0x4c, 0xbf, 0x15, 0xf5, + 0x6e, 0xcc, 0x18, 0x51, 0xa7, 0xe5, 0x1e, 0xb4, 0x27, 0xde, 0x76, 0xb4, 0x4b, 0x66, 0x3f, 0xf9, + 0xcc, 0x55, 0xe6, 0x73, 0xb1, 0x44, 0x99, 0xd6, 0x3d, 0xb3, 0x44, 0xd3, 0x2f, 0x39, 0xbd, 0x97, + 0x66, 0x0f, 0x2a, 0xad, 0x76, 0xa1, 0x95, 0x7f, 0xc4, 0xd1, 0xcc, 0x66, 0x3e, 0xed, 0x5c, 0xbe, + 0xde, 0xb9, 0xf7, 0x9c, 0x74, 0xbd, 0x67, 0x3d, 0xf3, 0xcc, 0x65, 0xb4, 0x05, 0xa0, 0x1a, 0x75, + 0x0f, 0x07, 0x89, 0xa3, 0xa7, 0x2e, 0x08, 0x12, 0x47, 0xcf, 0x68, 0xea, 0x1f, 0x00, 0xc8, 0xfe, + 0xda, 0x23, 0x31, 0x33, 0xae, 0x6b, 0x35, 0x26, 0x9a, 0xfa, 0x5e, 0x77, 0x7a, 0x60, 0x8a, 0x01, + 0xa2, 0xf4, 0x2a, 0x0c, 0x3e, 0x01, 0x48, 0xfb, 0x76, 0xcd, 0x60, 0xaa, 0x93, 0xbf, 0xc4, 0x07, + 0x8d, 0x6c, 0x97, 0x6e, 0x28, 0x5b, 0x67, 0x74, 0xee, 0x97, 0xb0, 0x68, 0x4f, 0x74, 0x61, 0xf9, + 0x60, 0x9b, 0x6c, 0xce, 0x7a, 0x53, 0x9d, 0x98, 0x71, 0x1f, 0x1a, 0xd9, 0xf6, 0x4b, 0x6b, 0x31, + 0xa3, 0x25, 0xeb, 0xe5, 0x5a, 0x30, 0xe3, 0x01, 0xb4, 0xf2, 0xad, 0x97, 0x0e, 0xa9, 0x99, 0x0d, + 0x59, 0x4f, 0x5d, 0x2c, 0x66, 0xc8, 0xdf, 0x05, 0x48, 0x5b, 0x34, 0xed, 0xbe, 0xa9, 0xa6, 0x6d, + 0x42, 0xea, 0x1e, 0xb4, 0x27, 0x5a, 0x2f, 0x6d, 0xf1, 0xec, 0x8e, 0xec, 0xb2, 0xbd, 0x9e, 0x69, + 0xa4, 0x74, 0x08, 0x4e, 0xb7, 0x62, 0x3a, 0x04, 0x67, 0x75, 0x5d, 0xdb, 0x50, 0x1f, 0x4c, 0xf3, + 0x18, 0xcc, 0xe5, 0x31, 0xab, 0x97, 0x7a, 0x0f, 0x20, 0x3d, 0xb6, 0xb4, 0x17, 0xa6, 0x0e, 0xb2, + 0x5e, 0x53, 0x5f, 0xfe, 0x4a, 0xba, 0x1d, 0x68, 0xe6, 0xee, 0x47, 0x74, 0xba, 0x9b, 0x75, 0x69, + 0x72, 0xd9, 0x21, 0x90, 0xbf, 0x4c, 0xd0, 0x2b, 0x38, 0xf3, 0x8a, 0xe1, 0xb2, 0x38, 0xce, 0x76, + 0x7e, 0x3a, 0x82, 0x66, 0x74, 0x83, 0xdf, 0x93, 0x57, 0xb2, 0xdd, 0x5d, 0x26, 0xaf, 0xcc, 0x68, + 0xfa, 0xe6, 0x32, 0xda, 0x87, 0xf6, 0x9e, 0x2e, 0xdc, 0x55, 0x53, 0xa1, 0xd7, 0x6f, 0xba, 0x89, + 0xea, 0xf5, 0x66, 0x0d, 0xa9, 0x75, 0xf9, 0x1c, 0x3a, 0x53, 0x0d, 0x85, 0xb1, 0x96, 0x5c, 0xc1, + 0xcf, 0xec, 0x34, 0xe6, 0xaa, 0x75, 0x00, 0x4b, 0x93, 0xfd, 0x84, 0xf1, 0x72, 0x12, 0x13, 0xb3, + 0xfa, 0x8c, 0xb9, 0xac, 0x3e, 0x84, 0xaa, 0xae, 0x11, 0x0d, 0xf5, 0xd4, 0x31, 0x51, 0x33, 0xce, + 0x9d, 0x7a, 0x5f, 0x84, 0x7c, 0x52, 0x7f, 0xa5, 0x21, 0x3f, 0x51, 0xa5, 0xf5, 0xd4, 0xcb, 0x44, + 0x42, 0x79, 0x1f, 0x2a, 0xaa, 0x0c, 0x33, 0x56, 0x92, 0xcd, 0x96, 0xa9, 0xca, 0x2e, 0x8b, 0xb0, + 0x3d, 0xc4, 0x32, 0xc5, 0x95, 0x16, 0x3a, 0x5d, 0x6f, 0xe9, 0x3d, 0x92, 0x1b, 0x51, 0x6b, 0xb1, + 0x05, 0x8d, 0x6c, 0x79, 0xa5, 0x97, 0x74, 0x46, 0xc9, 0x35, 0x4f, 0x93, 0xed, 0xf3, 0xa7, 0xdf, + 0xad, 0xbd, 0xf0, 0xb7, 0xef, 0xd6, 0x5e, 0xf8, 0xd5, 0xb3, 0xb5, 0xc2, 0xd3, 0x67, 0x6b, 0x85, + 0xbf, 0x3e, 0x5b, 0x2b, 0xfc, 0xf3, 0xd9, 0x5a, 0xe1, 0x27, 0x3f, 0xff, 0x0f, 0xff, 0x73, 0x45, + 0xe3, 0x80, 0x61, 0x1f, 0x6d, 0x8e, 0x31, 0x65, 0x99, 0xa1, 0xf0, 0x6c, 0x28, 0xff, 0x78, 0x95, + 0xf9, 0x3f, 0x16, 0xd7, 0xf2, 0x78, 0x51, 0xc0, 0xef, 0xfe, 0x3b, 0x00, 0x00, 0xff, 0xff, 0x77, + 0xb9, 0x27, 0x67, 0xdc, 0x25, 0x00, 0x00, } func (m *CreateContainerRequest) Marshal() (dAtA []byte, err error) { @@ -4822,6 +4992,155 @@ func (m *AddARPNeighborsRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) return len(dAtA) - i, nil } +func (m *GetIPTablesRequest) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *GetIPTablesRequest) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *GetIPTablesRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if m.IsIpv6 { + i-- + if m.IsIpv6 { + dAtA[i] = 1 + } else { + dAtA[i] = 0 + } + i-- + dAtA[i] = 0x8 + } + return len(dAtA) - i, nil +} + +func (m *GetIPTablesResponse) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *GetIPTablesResponse) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *GetIPTablesResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if len(m.Data) > 0 { + i -= len(m.Data) + copy(dAtA[i:], m.Data) + i = encodeVarintAgent(dAtA, i, uint64(len(m.Data))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + +func (m *SetIPTablesRequest) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *SetIPTablesRequest) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *SetIPTablesRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if len(m.Data) > 0 { + i -= len(m.Data) + copy(dAtA[i:], m.Data) + i = encodeVarintAgent(dAtA, i, uint64(len(m.Data))) + i-- + dAtA[i] = 0x12 + } + if m.IsIpv6 { + i-- + if m.IsIpv6 { + dAtA[i] = 1 + } else { + dAtA[i] = 0 + } + i-- + dAtA[i] = 0x8 + } + return len(dAtA) - i, nil +} + +func (m *SetIPTablesResponse) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *SetIPTablesResponse) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *SetIPTablesResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if len(m.Data) > 0 { + i -= len(m.Data) + copy(dAtA[i:], m.Data) + i = encodeVarintAgent(dAtA, i, uint64(len(m.Data))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + func (m *OnlineCPUMemRequest) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) @@ -6640,6 +6959,72 @@ func (m *AddARPNeighborsRequest) Size() (n int) { return n } +func (m *GetIPTablesRequest) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.IsIpv6 { + n += 2 + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *GetIPTablesResponse) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.Data) + if l > 0 { + n += 1 + l + sovAgent(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *SetIPTablesRequest) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.IsIpv6 { + n += 2 + } + l = len(m.Data) + if l > 0 { + n += 1 + l + sovAgent(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *SetIPTablesResponse) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.Data) + if l > 0 { + n += 1 + l + sovAgent(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + func (m *OnlineCPUMemRequest) Size() (n int) { if m == nil { return 0 @@ -7670,6 +8055,51 @@ func (this *AddARPNeighborsRequest) String() string { }, "") return s } +func (this *GetIPTablesRequest) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&GetIPTablesRequest{`, + `IsIpv6:` + fmt.Sprintf("%v", this.IsIpv6) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} +func (this *GetIPTablesResponse) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&GetIPTablesResponse{`, + `Data:` + fmt.Sprintf("%v", this.Data) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} +func (this *SetIPTablesRequest) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&SetIPTablesRequest{`, + `IsIpv6:` + fmt.Sprintf("%v", this.IsIpv6) + `,`, + `Data:` + fmt.Sprintf("%v", this.Data) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} +func (this *SetIPTablesResponse) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&SetIPTablesResponse{`, + `Data:` + fmt.Sprintf("%v", this.Data) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} func (this *OnlineCPUMemRequest) String() string { if this == nil { return "nil" @@ -7938,6 +8368,8 @@ type AgentServiceService interface { ListInterfaces(ctx context.Context, req *ListInterfacesRequest) (*Interfaces, error) ListRoutes(ctx context.Context, req *ListRoutesRequest) (*Routes, error) AddARPNeighbors(ctx context.Context, req *AddARPNeighborsRequest) (*types.Empty, error) + GetIPTables(ctx context.Context, req *GetIPTablesRequest) (*GetIPTablesResponse, error) + SetIPTables(ctx context.Context, req *SetIPTablesRequest) (*SetIPTablesResponse, error) GetMetrics(ctx context.Context, req *GetMetricsRequest) (*Metrics, error) CreateSandbox(ctx context.Context, req *CreateSandboxRequest) (*types.Empty, error) DestroySandbox(ctx context.Context, req *DestroySandboxRequest) (*types.Empty, error) @@ -8095,6 +8527,20 @@ func RegisterAgentServiceService(srv *github_com_containerd_ttrpc.Server, svc Ag } return svc.AddARPNeighbors(ctx, &req) }, + "GetIPTables": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { + var req GetIPTablesRequest + if err := unmarshal(&req); err != nil { + return nil, err + } + return svc.GetIPTables(ctx, &req) + }, + "SetIPTables": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { + var req SetIPTablesRequest + if err := unmarshal(&req); err != nil { + return nil, err + } + return svc.SetIPTables(ctx, &req) + }, "GetMetrics": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { var req GetMetricsRequest if err := unmarshal(&req); err != nil { @@ -8359,6 +8805,22 @@ func (c *agentServiceClient) AddARPNeighbors(ctx context.Context, req *AddARPNei return &resp, nil } +func (c *agentServiceClient) GetIPTables(ctx context.Context, req *GetIPTablesRequest) (*GetIPTablesResponse, error) { + var resp GetIPTablesResponse + if err := c.client.Call(ctx, "grpc.AgentService", "GetIPTables", req, &resp); err != nil { + return nil, err + } + return &resp, nil +} + +func (c *agentServiceClient) SetIPTables(ctx context.Context, req *SetIPTablesRequest) (*SetIPTablesResponse, error) { + var resp SetIPTablesResponse + if err := c.client.Call(ctx, "grpc.AgentService", "SetIPTables", req, &resp); err != nil { + return nil, err + } + return &resp, nil +} + func (c *agentServiceClient) GetMetrics(ctx context.Context, req *GetMetricsRequest) (*Metrics, error) { var resp Metrics if err := c.client.Call(ctx, "grpc.AgentService", "GetMetrics", req, &resp); err != nil { @@ -13746,6 +14208,352 @@ func (m *AddARPNeighborsRequest) Unmarshal(dAtA []byte) error { } return nil } +func (m *GetIPTablesRequest) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: GetIPTablesRequest: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: GetIPTablesRequest: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field IsIpv6", wireType) + } + var v int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + m.IsIpv6 = bool(v != 0) + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *GetIPTablesResponse) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: GetIPTablesResponse: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: GetIPTablesResponse: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAgent + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAgent + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *SetIPTablesRequest) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: SetIPTablesRequest: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: SetIPTablesRequest: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field IsIpv6", wireType) + } + var v int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + m.IsIpv6 = bool(v != 0) + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAgent + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAgent + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *SetIPTablesResponse) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: SetIPTablesResponse: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: SetIPTablesResponse: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAgent + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAgent + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} func (m *OnlineCPUMemRequest) Unmarshal(dAtA []byte) error { l := len(dAtA) iNdEx := 0 diff --git a/src/runtime/virtcontainers/pkg/mock/mock.go b/src/runtime/virtcontainers/pkg/mock/mock.go index 919f2e8a5..3b65e1152 100644 --- a/src/runtime/virtcontainers/pkg/mock/mock.go +++ b/src/runtime/virtcontainers/pkg/mock/mock.go @@ -244,3 +244,11 @@ func (p *HybridVSockTTRPCMockImp) ResizeVolume(ctx context.Context, req *pb.Resi func (p *HybridVSockTTRPCMockImp) PullImage(ctx context.Context, req *pb.PullImageRequest) (*gpb.Empty, error) { return &gpb.Empty{}, nil } + +func (p *HybridVSockTTRPCMockImp) GetIPTables(ctx context.Context, req *pb.GetIPTablesRequest) (*pb.GetIPTablesResponse, error) { + return &pb.GetIPTablesResponse{}, nil +} + +func (p *HybridVSockTTRPCMockImp) SetIPTables(ctx context.Context, req *pb.SetIPTablesRequest) (*pb.SetIPTablesResponse, error) { + return &pb.SetIPTablesResponse{}, nil +} diff --git a/src/runtime/virtcontainers/pkg/vcmock/sandbox.go b/src/runtime/virtcontainers/pkg/vcmock/sandbox.go index ed0072dc7..12710fb73 100644 --- a/src/runtime/virtcontainers/pkg/vcmock/sandbox.go +++ b/src/runtime/virtcontainers/pkg/vcmock/sandbox.go @@ -266,3 +266,11 @@ func (s *Sandbox) ResizeGuestVolume(ctx context.Context, path string, size uint6 func (s *Sandbox) PullImage(ctx context.Context, req *image.PullImageReq) (*image.PullImageResp, error) { return nil, nil } + +func (s *Sandbox) GetIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + return nil, nil +} + +func (s *Sandbox) SetIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + return nil +} diff --git a/src/runtime/virtcontainers/sandbox.go b/src/runtime/virtcontainers/sandbox.go index ba7d15360..d768793b7 100644 --- a/src/runtime/virtcontainers/sandbox.go +++ b/src/runtime/virtcontainers/sandbox.go @@ -2258,6 +2258,16 @@ func (s *Sandbox) GetAgentURL() (string, error) { return s.agent.getAgentURL() } +// GetIPTables will obtain the iptables from the guest +func (s *Sandbox) GetIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + return s.agent.getIPTables(ctx, isIPv6) +} + +// SetIPTables will set the iptables in the guest +func (s *Sandbox) SetIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + return s.agent.setIPTables(ctx, isIPv6, data) +} + // GuestVolumeStats return the filesystem stat of a given volume in the guest. func (s *Sandbox) GuestVolumeStats(ctx context.Context, volumePath string) ([]byte, error) { guestMountPath, err := s.guestMountPath(volumePath) diff --git a/src/tools/agent-ctl/src/client.rs b/src/tools/agent-ctl/src/client.rs index 2c6e0aa8a..7001649ab 100644 --- a/src/tools/agent-ctl/src/client.rs +++ b/src/tools/agent-ctl/src/client.rs @@ -165,6 +165,11 @@ static AGENT_CMDS: &[AgentCmd] = &[ st: ServiceType::Agent, fp: agent_cmd_sandbox_get_guest_details, }, + AgentCmd { + name: "GetIptables", + st: ServiceType::Agent, + fp: agent_cmd_sandbox_get_ip_tables, + }, AgentCmd { name: "GetMetrics", st: ServiceType::Agent, @@ -235,6 +240,11 @@ static AGENT_CMDS: &[AgentCmd] = &[ st: ServiceType::Agent, fp: agent_cmd_sandbox_set_guest_date_time, }, + AgentCmd { + name: "SetIptables", + st: ServiceType::Agent, + fp: agent_cmd_sandbox_set_ip_tables, + }, AgentCmd { name: "SignalProcess", st: ServiceType::Agent, @@ -1268,6 +1278,29 @@ fn agent_cmd_sandbox_get_guest_details( Ok(()) } +fn agent_cmd_sandbox_get_ip_tables( + ctx: &Context, + client: &AgentServiceClient, + _health: &HealthClient, + _options: &mut Options, + args: &str, +) -> Result<()> { + let req: GetIPTablesRequest = utils::make_request(args)?; + + let ctx = clone_context(ctx); + + debug!(sl!(), "sending request"; "request" => format!("{:?}", req)); + + let reply = client + .get_ip_tables(ctx, &req) + .map_err(|e| anyhow!("{:?}", e).context(ERR_API_FAILED))?; + + info!(sl!(), "response received"; + "response" => format!("{:?}", reply)); + + Ok(()) +} + fn agent_cmd_container_wait_process( ctx: &Context, client: &AgentServiceClient, @@ -1966,6 +1999,29 @@ fn agent_cmd_sandbox_set_guest_date_time( Ok(()) } +fn agent_cmd_sandbox_set_ip_tables( + ctx: &Context, + client: &AgentServiceClient, + _health: &HealthClient, + _options: &mut Options, + args: &str, +) -> Result<()> { + let req: SetIPTablesRequest = utils::make_request(args)?; + + let ctx = clone_context(ctx); + + debug!(sl!(), "sending request"; "request" => format!("{:?}", req)); + + let reply = client + .set_ip_tables(ctx, &req) + .map_err(|e| anyhow!(e).context(ERR_API_FAILED))?; + + info!(sl!(), "response received"; + "response" => format!("{:?}", reply)); + + Ok(()) +} + fn agent_cmd_sandbox_add_arp_neighbors( ctx: &Context, client: &AgentServiceClient, diff --git a/src/tools/runk/Cargo.lock b/src/tools/runk/Cargo.lock index f8628cbf7..a1691a796 100644 --- a/src/tools/runk/Cargo.lock +++ b/src/tools/runk/Cargo.lock @@ -985,7 +985,9 @@ dependencies = [ "serde_json", "slog", "slog-async", + "tabwriter", "tokio", + "users", ] [[package]] @@ -1166,6 +1168,15 @@ dependencies = [ "unicode-xid", ] +[[package]] +name = "tabwriter" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "36205cfc997faadcc4b0b87aaef3fbedafe20d38d4959a7ca6ff803564051111" +dependencies = [ + "unicode-width", +] + [[package]] name = "take_mut" version = "0.2.2" @@ -1348,12 +1359,28 @@ version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7e8820f5d777f6224dc4be3632222971ac30164d4a258d595640799554ebfd99" +[[package]] +name = "unicode-width" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ed742d4ea2bd1176e236172c8429aaf54486e7ac098db29ffe6529e0ce50973" + [[package]] name = "unicode-xid" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3" +[[package]] +name = "users" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24cc0f6d6f267b73e5a2cadf007ba8f9bc39c6a6f9666f8cf25ea809a153b032" +dependencies = [ + "libc", + "log", +] + [[package]] name = "version_check" version = "0.9.4" diff --git a/src/tools/runk/Cargo.toml b/src/tools/runk/Cargo.toml index 83aca82ca..973ae21c7 100644 --- a/src/tools/runk/Cargo.toml +++ b/src/tools/runk/Cargo.toml @@ -22,6 +22,8 @@ slog-async = "2.7.0" tokio = { version = "1.15.0", features = ["full"] } serde = { version = "1.0.133", features = ["derive"] } serde_json = "1.0.74" +users = "0.11.0" +tabwriter = "1.2.1" [workspace] members = [ diff --git a/src/tools/runk/Makefile b/src/tools/runk/Makefile index 580aa39a0..420de1b38 100644 --- a/src/tools/runk/Makefile +++ b/src/tools/runk/Makefile @@ -7,6 +7,7 @@ include ../../../utils.mk TARGET = runk TARGET_PATH = target/$(TRIPLE)/$(BUILD_TYPE)/$(TARGET) +AGENT_SOURCE_PATH = ../../agent # BINDIR is a directory for installing executable programs BINDIR := /usr/local/bin @@ -26,9 +27,14 @@ clean: vendor: cargo vendor -test: +test: test-runk test-agent + +test-runk: cargo test --all --target $(TRIPLE) -- --nocapture +test-agent: + make test -C $(AGENT_SOURCE_PATH) STANDARD_OCI_RUNTIME=yes + check: standard_rust_check .PHONY: \ diff --git a/src/tools/runk/README.md b/src/tools/runk/README.md index 9d18e27bc..10e24753f 100644 --- a/src/tools/runk/README.md +++ b/src/tools/runk/README.md @@ -149,6 +149,26 @@ $ sudo runk state test $ sudo runk delete test ``` +## Using `runk` from `Podman` + +`runk` can run containers using [`Podman`](https://github.com/containers/podman). + +First, install `Podman` from source code or package by following the +[`Podman` installation instructions](https://podman.io/getting-started/installation). + +### Running a container with `Podman` command line + +```bash +$ sudo podman --runtime /usr/local/bin/runk run -it --rm busybox sh +/ # +``` + +> **Note:** +> `runk` does not support some commands except +> [OCI standard operations](https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#operations) +> yet, so those commands do not work in `Podman`. Regarding commands currently +> implemented in `runk`, see the [Status of `runk`](#status-of-runk) section. + ## Using `runk` from `containerd` `runk` can run containers with the containerd runtime handler support on `containerd`. diff --git a/src/tools/runk/libcontainer/src/builder.rs b/src/tools/runk/libcontainer/src/builder.rs index 28bb73046..2ed8c0499 100644 --- a/src/tools/runk/libcontainer/src/builder.rs +++ b/src/tools/runk/libcontainer/src/builder.rs @@ -48,6 +48,16 @@ impl Container { } } + if let Some(process) = spec.process.as_ref() { + // runk always launches containers with detached mode, so users have to + // use a console socket with run or create operation when a terminal is used. + if process.terminal && self.console_socket.is_none() { + return Err(anyhow!( + "cannot allocate a pseudo-TTY without setting a console socket" + )); + } + } + Ok(ContainerContext { id: self.id, bundle: bundle_canon, @@ -66,7 +76,7 @@ impl Container { mod tests { use super::*; use crate::container::CONFIG_FILE_NAME; - use oci::Spec; + use oci::{self, Spec}; use std::{fs::File, path::PathBuf}; use tempfile::tempdir; @@ -118,4 +128,37 @@ mod tests { assert_eq!(test_ctx, ctx); } + + #[test] + fn test_create_ctx_tty_err() { + let bundle_dir = tempdir().unwrap(); + let config_file = bundle_dir.path().join(CONFIG_FILE_NAME); + + let mut spec = Spec::default(); + spec.process = Some(oci::Process::default()); + spec.process.as_mut().unwrap().terminal = true; + + let file = File::create(config_file).unwrap(); + serde_json::to_writer(&file, &spec).unwrap(); + + let test_data = TestData { + id: String::from("test"), + bundle: PathBuf::from(bundle_dir.into_path()), + root: PathBuf::from("test"), + console_socket: None, + spec: Spec::default(), + no_pivot_root: false, + }; + + let ctx = ContainerBuilder::default() + .id(test_data.id.clone()) + .bundle(test_data.bundle.clone()) + .root(test_data.root.clone()) + .console_socket(test_data.console_socket.clone()) + .build() + .unwrap() + .create_ctx(); + + assert!(ctx.is_err()); + } } diff --git a/src/tools/runk/src/commands/list.rs b/src/tools/runk/src/commands/list.rs new file mode 100644 index 000000000..dbd427755 --- /dev/null +++ b/src/tools/runk/src/commands/list.rs @@ -0,0 +1,69 @@ +// Copyright 2021-2022 Kata Contributors +// +// SPDX-License-Identifier: Apache-2.0 +// + +use super::state::get_container_state_name; +use anyhow::Result; +use libcontainer::status::{get_current_container_state, Status}; +use liboci_cli::List; +use oci::ContainerState; +use slog::{info, Logger}; +use std::{fs, os::unix::prelude::MetadataExt, path::Path}; +use std::{io, io::Write}; +use tabwriter::TabWriter; +use users::get_user_by_uid; + +pub fn run(_: List, root: &Path, logger: &Logger) -> Result<()> { + let mut content = String::new(); + for entry in fs::read_dir(root)? { + let entry = entry?; + // Possibly race with runk delete, so continue loop when any error occurs below + let metadata = match entry.metadata() { + Ok(metadata) => metadata, + Err(_) => continue, + }; + if !metadata.is_dir() { + continue; + } + let container_id = match entry.file_name().into_string() { + Ok(id) => id, + Err(_) => continue, + }; + let status = match Status::load(root, &container_id) { + Ok(status) => status, + Err(_) => continue, + }; + let state = match get_current_container_state(&status) { + Ok(state) => state, + Err(_) => continue, + }; + // Just like runc, pid of stopped container is 0 + let pid = match state { + ContainerState::Stopped => 0, + _ => status.pid, + }; + // May replace get_user_by_uid with getpwuid(3) + let owner = match get_user_by_uid(metadata.uid()) { + Some(user) => String::from(user.name().to_string_lossy()), + None => format!("#{}", metadata.uid()), + }; + content.push_str(&format!( + "{}\t{}\t{}\t{}\t{}\t{}\n", + container_id, + pid, + get_container_state_name(state), + status.bundle.display(), + status.created, + owner + )); + } + + let mut tab_writer = TabWriter::new(io::stdout()); + writeln!(&mut tab_writer, "ID\tPID\tSTATUS\tBUNDLE\tCREATED\tOWNER")?; + write!(&mut tab_writer, "{}", content)?; + tab_writer.flush()?; + + info!(&logger, "list command finished successfully"); + Ok(()) +} diff --git a/src/tools/runk/src/commands/mod.rs b/src/tools/runk/src/commands/mod.rs index 12017263d..216686506 100644 --- a/src/tools/runk/src/commands/mod.rs +++ b/src/tools/runk/src/commands/mod.rs @@ -6,6 +6,7 @@ pub mod create; pub mod delete; pub mod kill; +pub mod list; pub mod run; pub mod spec; pub mod start; diff --git a/src/tools/runk/src/main.rs b/src/tools/runk/src/main.rs index 8f464f438..0171a229f 100644 --- a/src/tools/runk/src/main.rs +++ b/src/tools/runk/src/main.rs @@ -78,6 +78,7 @@ async fn cmd_run(subcmd: SubCommand, root_path: &Path, logger: &Logger) -> Resul SubCommand::Common(cmd) => match cmd { CommonCmd::Run(run) => commands::run::run(run, root_path, logger).await, CommonCmd::Spec(spec) => commands::spec::run(spec, logger), + CommonCmd::List(list) => commands::list::run(list, root_path, logger), _ => { return Err(anyhow!("command is not implemented yet")); } diff --git a/tools/osbuilder/rootfs-builder/alpine/config.sh b/tools/osbuilder/rootfs-builder/alpine/config.sh index 105ecd8fb..1df524333 100644 --- a/tools/osbuilder/rootfs-builder/alpine/config.sh +++ b/tools/osbuilder/rootfs-builder/alpine/config.sh @@ -13,7 +13,7 @@ BASE_PACKAGES="alpine-base" # See a list of mirrors at http://nl.alpinelinux.org/alpine/MIRRORS.txt MIRROR=https://dl-5.alpinelinux.org/alpine -PACKAGES="" +PACKAGES="iptables ip6tables" # Init process must be one of {systemd,kata-agent} INIT_PROCESS=kata-agent diff --git a/tools/osbuilder/rootfs-builder/centos/config.sh b/tools/osbuilder/rootfs-builder/centos/config.sh index d9d51e5d7..7226da047 100644 --- a/tools/osbuilder/rootfs-builder/centos/config.sh +++ b/tools/osbuilder/rootfs-builder/centos/config.sh @@ -5,7 +5,7 @@ OS_NAME=centos OS_VERSION=${OS_VERSION:-stream9} -PACKAGES=chrony +PACKAGES="chrony iptables" [ "$AGENT_INIT" = no ] && PACKAGES+=" systemd" [ "$SECCOMP" = yes ] && PACKAGES+=" libseccomp" diff --git a/tools/osbuilder/rootfs-builder/clearlinux/config.sh b/tools/osbuilder/rootfs-builder/clearlinux/config.sh index cf8497624..e447407ac 100644 --- a/tools/osbuilder/rootfs-builder/clearlinux/config.sh +++ b/tools/osbuilder/rootfs-builder/clearlinux/config.sh @@ -15,12 +15,12 @@ clr_url="https://download.clearlinux.org" BASE_URL="${clr_url}/releases/${OS_VERSION}/${REPO_NAME}/${ARCH}/os/" -PACKAGES="libudev0-shim kmod-bin" +PACKAGES="libudev0-shim kmod-bin iptables-bin" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" = "no" ] && PACKAGES+=" systemd chrony iptables-bin util-linux-bin" || true +[ "$AGENT_INIT" = "no" ] && PACKAGES+=" systemd chrony util-linux-bin" || true # Init process must be one of {systemd,kata-agent} INIT_PROCESS=systemd diff --git a/tools/osbuilder/rootfs-builder/debian/config.sh b/tools/osbuilder/rootfs-builder/debian/config.sh index f87370921..203cad9f9 100644 --- a/tools/osbuilder/rootfs-builder/debian/config.sh +++ b/tools/osbuilder/rootfs-builder/debian/config.sh @@ -8,7 +8,7 @@ OS_VERSION=${OS_VERSION:-10.11} # Set OS_NAME to the desired debian "codename" OS_NAME=${OS_NAME:-"stretch"} -PACKAGES="systemd coreutils init chrony kmod" +PACKAGES="systemd coreutils init iptables chrony kmod" # NOTE: Re-using ubuntu rootfs configuration, see 'ubuntu' folder for full content. source $script_dir/ubuntu/$CONFIG_SH diff --git a/tools/osbuilder/rootfs-builder/ubuntu/config.sh b/tools/osbuilder/rootfs-builder/ubuntu/config.sh index d9e249f8d..e139856ca 100644 --- a/tools/osbuilder/rootfs-builder/ubuntu/config.sh +++ b/tools/osbuilder/rootfs-builder/ubuntu/config.sh @@ -5,7 +5,7 @@ OS_NAME=ubuntu # This should be Ubuntu's code name, e.g. "focal" (Focal Fossa) for 20.04 OS_VERSION=${OS_VERSION:-focal} -PACKAGES=chrony +PACKAGES="chrony iptables" [ "$AGENT_INIT" = no ] && PACKAGES+=" init" [ "$SECCOMP" = yes ] && PACKAGES+=" libseccomp2" [ "$SKOPEO" = yes ] && PACKAGES+=" libgpgme11" diff --git a/tools/packaging/Makefile b/tools/packaging/Makefile index a2668ff76..f8444e8f1 100644 --- a/tools/packaging/Makefile +++ b/tools/packaging/Makefile @@ -29,6 +29,6 @@ snap: $(YQ) @if [ "$$(cat $(VERSION_FILE))" != "$$($(YQ) r $(SNAPCRAFT_FILE) version)" ]; then \ >&2 echo "Warning: $(SNAPCRAFT_FILE) version is different to upstream $(VERSION_FILE) file"; \ fi - snapcraft -d + snapcraft -d --destructive-mode .PHONY: test-static-build snap diff --git a/tools/packaging/README.md b/tools/packaging/README.md index 19d0978d4..8a31c0a80 100644 --- a/tools/packaging/README.md +++ b/tools/packaging/README.md @@ -14,7 +14,7 @@ running Kubernetes Cluster very straightforward. ## Build a snap package -See [the snap documentation](../../snap). +See [the snap documentation](../../snap/local). ## Build static binaries