From 60823abb9c7f46c0eee610a00cdf878d6abbbb68 Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Thu, 26 May 2022 13:44:20 +0100 Subject: [PATCH 01/35] docs: Move snap README Move the snap README to a subdirectory to resolve the warning given by `snapcraft` (folded and reformatted slightly for clarity): ``` The 'snap' directory is meant specifically for snapcraft, but it contains the following non-snapcraft-related paths, which is unsupported and will cause unexpected behavior: - README.md If you must store these files within the 'snap' directory, move them to 'snap/local', which is ignored by snapcraft. ``` Signed-off-by: James O. D. Hunt --- README.md | 2 +- snap/{ => local}/README.md | 10 +++++----- tools/packaging/README.md | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) rename snap/{ => local}/README.md (89%) diff --git a/README.md b/README.md index 3cac62400..90a5c9209 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,7 @@ The table below lists the remaining parts of the project: Kata Containers is now [available natively for most distributions](docs/install/README.md#packaged-installation-methods). -However, packaging scripts and metadata are still used to generate snap and GitHub releases. See +However, packaging scripts and metadata are still used to generate [snap](snap/local) and GitHub releases. See the [components](#components) section for further details. ## Glossary of Terms diff --git a/snap/README.md b/snap/local/README.md similarity index 89% rename from snap/README.md rename to snap/local/README.md index 3729542ec..95eef59fb 100644 --- a/snap/README.md +++ b/snap/local/README.md @@ -76,12 +76,12 @@ then a new configuration file can be [created](#configure-kata-containers) and [configured][7]. [1]: https://docs.snapcraft.io/snaps/intro -[2]: ../docs/design/architecture/README.md#root-filesystem-image +[2]: ../../docs/design/architecture/README.md#root-filesystem-image [3]: https://docs.snapcraft.io/reference/confinement#classic [4]: https://github.com/kata-containers/runtime#configuration [5]: https://docs.docker.com/engine/reference/commandline/dockerd -[6]: ../docs/install/docker/ubuntu-docker-install.md -[7]: ../docs/Developer-Guide.md#configure-to-use-initrd-or-rootfs-image +[6]: ../../docs/install/docker/ubuntu-docker-install.md +[7]: ../../docs/Developer-Guide.md#configure-to-use-initrd-or-rootfs-image [8]: https://snapcraft.io/kata-containers -[9]: ../docs/Developer-Guide.md#run-kata-containers-with-docker -[10]: ../docs/Developer-Guide.md#run-kata-containers-with-kubernetes +[9]: ../../docs/Developer-Guide.md#run-kata-containers-with-docker +[10]: ../../docs/Developer-Guide.md#run-kata-containers-with-kubernetes diff --git a/tools/packaging/README.md b/tools/packaging/README.md index 19d0978d4..8a31c0a80 100644 --- a/tools/packaging/README.md +++ b/tools/packaging/README.md @@ -14,7 +14,7 @@ running Kubernetes Cluster very straightforward. ## Build a snap package -See [the snap documentation](../../snap). +See [the snap documentation](../../snap/local). ## Build static binaries From cb2b30970d1b62d9aa49ba77fee1bb56b6179e44 Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Thu, 26 May 2022 14:08:18 +0100 Subject: [PATCH 02/35] snap: Build using destructive mode Destructive mode is required to build the Kata Containers snap. See: ``` .github/workflows/snap-release.yaml .github/workflows/snap.yaml ``` Hence, update the last file that we forgot to update with `--destructive-mode`. Signed-off-by: James O. D. Hunt --- tools/packaging/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/packaging/Makefile b/tools/packaging/Makefile index a2668ff76..f8444e8f1 100644 --- a/tools/packaging/Makefile +++ b/tools/packaging/Makefile @@ -29,6 +29,6 @@ snap: $(YQ) @if [ "$$(cat $(VERSION_FILE))" != "$$($(YQ) r $(SNAPCRAFT_FILE) version)" ]; then \ >&2 echo "Warning: $(SNAPCRAFT_FILE) version is different to upstream $(VERSION_FILE) file"; \ fi - snapcraft -d + snapcraft -d --destructive-mode .PHONY: test-static-build snap From 56591804b38621c86b4bd57feee335729c303aa0 Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Thu, 26 May 2022 14:10:41 +0100 Subject: [PATCH 03/35] docs: Improve snap build instructions Make it clearer how to build the snap package manually. Signed-off-by: James O. D. Hunt --- snap/local/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/snap/local/README.md b/snap/local/README.md index 95eef59fb..60564f4b2 100644 --- a/snap/local/README.md +++ b/snap/local/README.md @@ -22,10 +22,10 @@ $ sudo snap install kata-containers --classic ## Build and install snap image -Run next command at the root directory of the packaging repository. +Run the command below which will use the packaging Makefile to build the snap image: ```sh -$ make snap +$ make -C tools/packaging snap ``` To install the resulting snap image, snap must be put in [classic mode][3] and the From 5205efd9b4783ab1246b8f0a03739f91cf367106 Mon Sep 17 00:00:00 2001 From: Manabu Sugimoto Date: Mon, 30 May 2022 15:55:58 +0900 Subject: [PATCH 04/35] runk: Add Podman guide in README runk can launch containers using Podman, so add the guide in README. Fixes: #4338 Signed-off-by: Manabu Sugimoto --- src/tools/runk/README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/src/tools/runk/README.md b/src/tools/runk/README.md index 9d18e27bc..10e24753f 100644 --- a/src/tools/runk/README.md +++ b/src/tools/runk/README.md @@ -149,6 +149,26 @@ $ sudo runk state test $ sudo runk delete test ``` +## Using `runk` from `Podman` + +`runk` can run containers using [`Podman`](https://github.com/containers/podman). + +First, install `Podman` from source code or package by following the +[`Podman` installation instructions](https://podman.io/getting-started/installation). + +### Running a container with `Podman` command line + +```bash +$ sudo podman --runtime /usr/local/bin/runk run -it --rm busybox sh +/ # +``` + +> **Note:** +> `runk` does not support some commands except +> [OCI standard operations](https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#operations) +> yet, so those commands do not work in `Podman`. Regarding commands currently +> implemented in `runk`, see the [Status of `runk`](#status-of-runk) section. + ## Using `runk` from `containerd` `runk` can run containers with the containerd runtime handler support on `containerd`. From 648b8d0aec83605ad4599867cbec0977837fc6d3 Mon Sep 17 00:00:00 2001 From: Manabu Sugimoto Date: Fri, 27 May 2022 12:19:00 +0900 Subject: [PATCH 05/35] runk: Return error when tty is used without console socket runk always launches containers with detached mode, so users have to use a console socket with run or create operation when a terminal is used. If users set `terminal` to `true` in `config.json` and try to launch a container without specifying a console socket, runk returns an error with a message early. Fixes: #4324 Signed-off-by: Manabu Sugimoto --- src/tools/runk/libcontainer/src/builder.rs | 45 +++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/src/tools/runk/libcontainer/src/builder.rs b/src/tools/runk/libcontainer/src/builder.rs index 738c639ae..ddbb7bd68 100644 --- a/src/tools/runk/libcontainer/src/builder.rs +++ b/src/tools/runk/libcontainer/src/builder.rs @@ -48,6 +48,16 @@ impl Container { } } + if let Some(process) = spec.process.as_ref() { + // runk always launches containers with detached mode, so users have to + // use a console socket with run or create operation when a terminal is used. + if process.terminal && self.console_socket.is_none() { + return Err(anyhow!( + "cannot allocate a pseudo-TTY without setting a console socket" + )); + } + } + Ok(ContainerContext { id: self.id, bundle: self.bundle, @@ -66,7 +76,7 @@ impl Container { mod tests { use super::*; use crate::container::CONFIG_FILE_NAME; - use oci::Spec; + use oci::{self, Spec}; use std::{fs::File, path::PathBuf}; use tempfile::tempdir; @@ -118,4 +128,37 @@ mod tests { assert_eq!(test_ctx, ctx); } + + #[test] + fn test_create_ctx_tty_err() { + let bundle_dir = tempdir().unwrap(); + let config_file = bundle_dir.path().join(CONFIG_FILE_NAME); + + let mut spec = Spec::default(); + spec.process = Some(oci::Process::default()); + spec.process.as_mut().unwrap().terminal = true; + + let file = File::create(config_file).unwrap(); + serde_json::to_writer(&file, &spec).unwrap(); + + let test_data = TestData { + id: String::from("test"), + bundle: PathBuf::from(bundle_dir.into_path()), + root: PathBuf::from("test"), + console_socket: None, + spec: Spec::default(), + no_pivot_root: false, + }; + + let ctx = ContainerBuilder::default() + .id(test_data.id.clone()) + .bundle(test_data.bundle.clone()) + .root(test_data.root.clone()) + .console_socket(test_data.console_socket.clone()) + .build() + .unwrap() + .create_ctx(); + + assert!(ctx.is_err()); + } } From 6ecea84bc5cee60497453f89afa7e9ba25d525a1 Mon Sep 17 00:00:00 2001 From: Xuewei Niu Date: Mon, 23 May 2022 17:23:54 +0800 Subject: [PATCH 06/35] rustjail: get home dir using nix crate Get user's home dir using `nix::unistd` crate instead of `utils` crate, and remove useless code from agent. Fixes: #4209 Signed-off-by: Xuewei Niu --- src/agent/rustjail/src/container.rs | 19 +++-- src/agent/rustjail/src/lib.rs | 1 - src/agent/rustjail/src/utils.rs | 120 ---------------------------- 3 files changed, 11 insertions(+), 129 deletions(-) delete mode 100644 src/agent/rustjail/src/utils.rs diff --git a/src/agent/rustjail/src/container.rs b/src/agent/rustjail/src/container.rs index 37ffa091d..4bfb83068 100644 --- a/src/agent/rustjail/src/container.rs +++ b/src/agent/rustjail/src/container.rs @@ -42,7 +42,7 @@ use nix::pty; use nix::sched::{self, CloneFlags}; use nix::sys::signal::{self, Signal}; use nix::sys::stat::{self, Mode}; -use nix::unistd::{self, fork, ForkResult, Gid, Pid, Uid}; +use nix::unistd::{self, fork, ForkResult, Gid, Pid, Uid, User}; use std::os::unix::fs::MetadataExt; use std::os::unix::io::AsRawFd; @@ -64,8 +64,6 @@ use rlimit::{setrlimit, Resource, Rlim}; use tokio::io::AsyncBufReadExt; use tokio::sync::Mutex; -use crate::utils; - pub const EXEC_FIFO_FILENAME: &str = "exec.fifo"; const INIT: &str = "INIT"; @@ -665,12 +663,17 @@ fn do_init_child(cwfd: RawFd) -> Result<()> { } } - // set the "HOME" env getting from "/etc/passwd", if - // there's no uid entry in /etc/passwd, set "/" as the - // home env. if env::var_os(HOME_ENV_KEY).is_none() { - let home_dir = utils::home_dir(guser.uid).unwrap_or_else(|_| String::from("/")); - env::set_var(HOME_ENV_KEY, home_dir); + // try to set "HOME" env by uid + if let Ok(Some(user)) = User::from_uid(Uid::from_raw(guser.uid)) { + if let Ok(user_home_dir) = user.dir.into_os_string().into_string() { + env::set_var(HOME_ENV_KEY, user_home_dir); + } + } + // set default home dir as "/" if "HOME" env is still empty + if env::var_os(HOME_ENV_KEY).is_none() { + env::set_var(HOME_ENV_KEY, String::from("/")); + } } let exec_file = Path::new(&args[0]); diff --git a/src/agent/rustjail/src/lib.rs b/src/agent/rustjail/src/lib.rs index dc7ff6cdc..dafac6381 100644 --- a/src/agent/rustjail/src/lib.rs +++ b/src/agent/rustjail/src/lib.rs @@ -41,7 +41,6 @@ pub mod seccomp; pub mod specconv; pub mod sync; pub mod sync_with_async; -pub mod utils; pub mod validator; use std::collections::HashMap; diff --git a/src/agent/rustjail/src/utils.rs b/src/agent/rustjail/src/utils.rs deleted file mode 100644 index cf69c77a2..000000000 --- a/src/agent/rustjail/src/utils.rs +++ /dev/null @@ -1,120 +0,0 @@ -// Copyright (c) 2021 Ant Group -// -// SPDX-License-Identifier: Apache-2.0 -// -use anyhow::{anyhow, Context, Result}; -use libc::gid_t; -use libc::uid_t; -use std::fs::File; -use std::io::{BufRead, BufReader}; - -const PASSWD_FILE: &str = "/etc/passwd"; - -// An entry from /etc/passwd -#[derive(Debug, PartialEq, PartialOrd)] -pub struct PasswdEntry { - // username - pub name: String, - // user password - pub passwd: String, - // user id - pub uid: uid_t, - // group id - pub gid: gid_t, - // user Information - pub gecos: String, - // home directory - pub dir: String, - // User's Shell - pub shell: String, -} - -// get an entry for a given `uid` from `/etc/passwd` -fn get_entry_by_uid(uid: uid_t, path: &str) -> Result { - let file = File::open(path).with_context(|| format!("open file {}", path))?; - let mut reader = BufReader::new(file); - - let mut line = String::new(); - loop { - line.clear(); - match reader.read_line(&mut line) { - Ok(0) => return Err(anyhow!(format!("file {} is empty", path))), - Ok(_) => (), - Err(e) => { - return Err(anyhow!(format!( - "failed to read file {} with {:?}", - path, e - ))) - } - } - - if line.starts_with('#') { - continue; - } - - let parts: Vec<&str> = line.split(':').map(|part| part.trim()).collect(); - if parts.len() != 7 { - continue; - } - - match parts[2].parse() { - Err(_e) => continue, - Ok(new_uid) => { - if uid != new_uid { - continue; - } - - let entry = PasswdEntry { - name: parts[0].to_string(), - passwd: parts[1].to_string(), - uid: new_uid, - gid: parts[3].parse().unwrap_or(0), - gecos: parts[4].to_string(), - dir: parts[5].to_string(), - shell: parts[6].to_string(), - }; - - return Ok(entry); - } - } - } -} - -pub fn home_dir(uid: uid_t) -> Result { - get_entry_by_uid(uid, PASSWD_FILE).map(|entry| entry.dir) -} - -#[cfg(test)] -mod tests { - use super::*; - use std::io::Write; - use tempfile::Builder; - - #[test] - fn test_get_entry_by_uid() { - let tmpdir = Builder::new().tempdir().unwrap(); - let tmpdir_path = tmpdir.path().to_str().unwrap(); - let temp_passwd = format!("{}/passwd", tmpdir_path); - - let mut tempf = File::create(temp_passwd.as_str()).unwrap(); - let passwd_entries = "root:x:0:0:root:/root0:/bin/bash -root:x:1:0:root:/root1:/bin/bash -#root:x:1:0:root:/rootx:/bin/bash -root:x:2:0:root:/root2:/bin/bash -root:x:3:0:root:/root3 -root:x:3:0:root:/root3:/bin/bash"; - writeln!(tempf, "{}", passwd_entries).unwrap(); - - let entry = get_entry_by_uid(0, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root0"); - - let entry = get_entry_by_uid(1, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root1"); - - let entry = get_entry_by_uid(2, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root2"); - - let entry = get_entry_by_uid(3, temp_passwd.as_str()).unwrap(); - assert_eq!(entry.dir.as_str(), "/root3"); - } -} From 6dbce7c3dee55a25354d535356d8dae5850006da Mon Sep 17 00:00:00 2001 From: Manabu Sugimoto Date: Tue, 31 May 2022 16:25:16 +0900 Subject: [PATCH 07/35] agent: Remove unused import in console test Remove some unused imports in console test module used by runk's test. Fixes: #4351 Signed-off-by: Manabu Sugimoto --- src/agent/rustjail/src/console.rs | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/agent/rustjail/src/console.rs b/src/agent/rustjail/src/console.rs index 23cf6e840..52e33f392 100644 --- a/src/agent/rustjail/src/console.rs +++ b/src/agent/rustjail/src/console.rs @@ -58,10 +58,7 @@ pub fn setup_master_console(socket_fd: RawFd) -> Result<()> { #[cfg(test)] mod tests { use super::*; - use crate::skip_if_not_root; - use std::fs::File; use std::os::unix::net::UnixListener; - use std::path::PathBuf; use tempfile::{self, tempdir}; const CONSOLE_SOCKET: &str = "console-socket"; From c7b3941c9653b9860c932134dc391586ba921e8a Mon Sep 17 00:00:00 2001 From: Manabu Sugimoto Date: Tue, 31 May 2022 16:24:39 +0900 Subject: [PATCH 08/35] runk: Enable test for the agent built with standard-oci-runtime feature This enables tests for the kata-agent for runk that is built with standard-oci-runtime feature in CI. Fixes: #4351 Signed-off-by: Manabu Sugimoto --- src/tools/runk/Makefile | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/tools/runk/Makefile b/src/tools/runk/Makefile index 580aa39a0..420de1b38 100644 --- a/src/tools/runk/Makefile +++ b/src/tools/runk/Makefile @@ -7,6 +7,7 @@ include ../../../utils.mk TARGET = runk TARGET_PATH = target/$(TRIPLE)/$(BUILD_TYPE)/$(TARGET) +AGENT_SOURCE_PATH = ../../agent # BINDIR is a directory for installing executable programs BINDIR := /usr/local/bin @@ -26,9 +27,14 @@ clean: vendor: cargo vendor -test: +test: test-runk test-agent + +test-runk: cargo test --all --target $(TRIPLE) -- --nocapture +test-agent: + make test -C $(AGENT_SOURCE_PATH) STANDARD_OCI_RUNTIME=yes + check: standard_rust_check .PHONY: \ From 78d45b434fe415ee28dbb8dd7ceb8d25a2d62028 Mon Sep 17 00:00:00 2001 From: Bin Liu Date: Tue, 31 May 2022 23:36:14 +0800 Subject: [PATCH 09/35] agent: return mount file content if parse mountinfo failed Include mount file content in error message when parsing mountinfo failed for debug. Fixes: #4246, #4103 Signed-off-by: Bin Liu --- src/agent/src/mount.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/agent/src/mount.rs b/src/agent/src/mount.rs index d4729c881..ff24f4957 100644 --- a/src/agent/src/mount.rs +++ b/src/agent/src/mount.rs @@ -859,8 +859,9 @@ pub fn get_mount_fs_type_from_file(mount_file: &str, mount_point: &str) -> Resul } Err(anyhow!( - "failed to find FS type for mount point {}", - mount_point + "failed to find FS type for mount point {}, mount file content: {:?}", + mount_point, + fs::read_to_string(mount_file) )) } From 38ebbc705bb786772839af6bb82b2b80e349b666 Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Thu, 21 Apr 2022 21:19:07 -0700 Subject: [PATCH 10/35] proto: update to add set/get iptables Update the agent protocol definition to introduce support for setting and getting iptables from the guest. Signed-off-by: Eric Ernst --- src/libs/Cargo.lock | 618 +++++++++++++++++++++++++- src/libs/Cargo.toml | 1 + src/libs/protocols/protos/agent.proto | 24 + 3 files changed, 638 insertions(+), 5 deletions(-) diff --git a/src/libs/Cargo.lock b/src/libs/Cargo.lock index ad9d8672c..fc7fc5da5 100644 --- a/src/libs/Cargo.lock +++ b/src/libs/Cargo.lock @@ -1,11 +1,30 @@ # This file is automatically @generated by Cargo. # It is not intended for manual editing. +version = 3 + +[[package]] +name = "anyhow" +version = "1.0.57" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08f9b8508dccb7687a1d6c4ce66b2b0ecef467c94667de27d8d7fe1f8d2a9cdc" + [[package]] name = "arc-swap" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c5d78ce20460b82d3fa150275ed9d55e21064fc7951177baacf86a145c4a4b1f" +[[package]] +name = "async-trait" +version = "0.1.53" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed6aa3524a2dfcf9fe180c51eae2b58738348d819517ceadf95789c51fff7600" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "autocfg" version = "1.0.1" @@ -14,9 +33,37 @@ checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a" [[package]] name = "bitflags" -version = "1.3.2" +version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" +checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693" + +[[package]] +name = "byteorder" +version = "1.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610" + +[[package]] +name = "bytes" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "206fdffcfa2df7cbe15601ef46c813fce0965eb3286db6b56c583b814b51c81c" +dependencies = [ + "byteorder", + "iovec", +] + +[[package]] +name = "bytes" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4872d67bab6358e59559027aa3b9157c53d9358c51423c17554809a8858e0f8" + +[[package]] +name = "cc" +version = "1.0.73" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2fff2a6927b3bb87f9595d67196a70493f627687a71d87a0d692242c33f58c11" [[package]] name = "cfg-if" @@ -57,6 +104,23 @@ dependencies = [ "lazy_static", ] +[[package]] +name = "derive-new" +version = "0.5.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3418329ca0ad70234b9735dc4ceed10af4df60eff9c8e7b06cb5e520d92c3535" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "either" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e78d4f1cc4ae33bbfc157ed5d5a5ef3bc29227303d595861deb238fcec4e9457" + [[package]] name = "fastrand" version = "1.6.0" @@ -66,6 +130,126 @@ dependencies = [ "instant", ] +[[package]] +name = "fixedbitset" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37ab347416e802de484e4d03c7316c48f1ecb56574dfd4a46a80f173ce1de04d" + +[[package]] +name = "futures" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f73fe65f54d1e12b726f517d3e2135ca3125a437b6d998caf1962961f7172d9e" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3083ce4b914124575708913bca19bfe887522d6e2e6d0952943f5eac4a74010" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c09fd04b7e4073ac7156a9539b57a484a8ea920f79c7c675d05d289ab6110d3" + +[[package]] +name = "futures-executor" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9420b90cfa29e327d0429f19be13e7ddb68fa1cccb09d65e5706b8c7a749b8a6" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc4045962a5a5e935ee2fdedaa4e08284547402885ab326734432bed5d12966b" + +[[package]] +name = "futures-macro" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33c1e13800337f4d4d7a316bf45a567dbcb6ffe087f16424852d97e97a91f512" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "futures-sink" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "21163e139fa306126e6eedaf49ecdb4588f939600f0b1e770f4205ee4b7fa868" + +[[package]] +name = "futures-task" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c66a976bf5909d801bbef33416c41372779507e7a6b3a5e25e4749c58f776a" + +[[package]] +name = "futures-util" +version = "0.3.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d8b7abd5d659d9b90c8cba917f6ec750a74e2dc23902ef9cd4cc8c8b22e6036a" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "pin-utils", + "slab", +] + +[[package]] +name = "hashbrown" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab5ef0d4909ef3724cc8cce6ccc8572c5c817592e9285f5464f8e86f8bd3726e" + +[[package]] +name = "heck" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d621efb26863f0e9924c6ac577e8275e5e6b77455db64ffa6c65c904e9e132c" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "indexmap" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f647032dfaa1f8b6dc29bd3edb7bbef4861b8b8007ebb118d6db284fd59f6ee" +dependencies = [ + "autocfg", + "hashbrown", +] + [[package]] name = "instant" version = "0.1.12" @@ -75,6 +259,24 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "iovec" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b2b3ea6ff95e175473f8ffe6a7eb7c00d054240321b84c57051175fe3c1e075e" +dependencies = [ + "libc", +] + +[[package]] +name = "itertools" +version = "0.10.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9a9d19fa1e79b6215ff29b9d6880b706147f16e9b1dbb1e4e5947b5b02bc5e3" +dependencies = [ + "either", +] + [[package]] name = "itoa" version = "1.0.1" @@ -89,9 +291,18 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" [[package]] name = "libc" -version = "0.2.113" +version = "0.2.124" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eef78b64d87775463c549fbd80e19249ef436ea3bf1de2a1eb7e717ec7fab1e9" +checksum = "21a41fed9d98f27ab1c6d161da622a4fa35e8a54a8adc24bbf3ddd0ef70b0e50" + +[[package]] +name = "log" +version = "0.4.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6389c490849ff5bc16be905ae24bc913a9c8892e19b2341dbc175e14c341c2b8" +dependencies = [ + "cfg-if", +] [[package]] name = "logging" @@ -105,6 +316,85 @@ dependencies = [ "tempfile", ] +[[package]] +name = "memchr" +version = "2.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "308cc39be01b73d0d18f82a0e7b2a3df85245f84af96fdddc5d202d27e47b86a" + +[[package]] +name = "memoffset" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5aa361d4faea93603064a027415f07bd8e1d5c88c9fbf68bf56a285428fd79ce" +dependencies = [ + "autocfg", +] + +[[package]] +name = "mio" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52da4364ffb0e4fe33a9841a98a3f3014fb964045ce4f7a45a398243c8d6b0c9" +dependencies = [ + "libc", + "log", + "miow", + "ntapi", + "wasi 0.11.0+wasi-snapshot-preview1", + "winapi", +] + +[[package]] +name = "miow" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9f1c5b025cda876f66ef43a113f91ebc9f4ccef34843000e0adf6ebbab84e21" +dependencies = [ + "winapi", +] + +[[package]] +name = "multimap" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5ce46fe64a9d73be07dcbe690a38ce1b293be448fd8ce1e6c1b8062c9f72c6a" + +[[package]] +name = "nix" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f5e06129fb611568ef4e868c14b326274959aa70ff7776e9d55323531c374945" +dependencies = [ + "bitflags", + "cc", + "cfg-if", + "libc", + "memoffset", +] + +[[package]] +name = "nix" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f866317acbd3a240710c63f065ffb1e4fd466259045ccb504130b7f668f35c6" +dependencies = [ + "bitflags", + "cc", + "cfg-if", + "libc", + "memoffset", +] + +[[package]] +name = "ntapi" +version = "0.3.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c28774a7fd2fbb4f0babd8237ce554b73af68021b5f695a3cebd6c59bac0980f" +dependencies = [ + "winapi", +] + [[package]] name = "num-integer" version = "0.1.44" @@ -130,6 +420,138 @@ version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "da32515d9f6e6e489d7bc9d84c71b060db7247dc035bbe44eac88cf87486d8d5" +[[package]] +name = "petgraph" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "467d164a6de56270bd7c4d070df81d07beace25012d5103ced4e9ff08d6afdb7" +dependencies = [ + "fixedbitset", + "indexmap", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e280fbe77cc62c91527259e9442153f4688736748d24660126286329742b4c6c" + +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + +[[package]] +name = "proc-macro2" +version = "1.0.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec757218438d5fda206afc041538b2f6d889286160d649a86a24d37e1235afd1" +dependencies = [ + "unicode-xid", +] + +[[package]] +name = "prost" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de5e2533f59d08fcf364fd374ebda0692a70bd6d7e66ef97f306f45c6c5d8020" +dependencies = [ + "bytes 1.1.0", + "prost-derive", +] + +[[package]] +name = "prost-build" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "355f634b43cdd80724ee7848f95770e7e70eefa6dcf14fea676216573b8fd603" +dependencies = [ + "bytes 1.1.0", + "heck", + "itertools", + "log", + "multimap", + "petgraph", + "prost", + "prost-types", + "tempfile", + "which", +] + +[[package]] +name = "prost-derive" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "600d2f334aa05acb02a755e217ef1ab6dea4d51b58b7846588b747edec04efba" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "prost-types" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "603bbd6394701d13f3f25aada59c7de9d35a6a5887cfc156181234a44002771b" +dependencies = [ + "bytes 1.1.0", + "prost", +] + +[[package]] +name = "protobuf" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e86d370532557ae7573551a1ec8235a0f8d6cb276c7c9e6aa490b511c447485" +dependencies = [ + "serde", + "serde_derive", +] + +[[package]] +name = "protobuf-codegen" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de113bba758ccf2c1ef816b127c958001b7831136c9bc3f8e9ec695ac4e82b0c" +dependencies = [ + "protobuf", +] + +[[package]] +name = "protobuf-codegen-pure" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d1a4febc73bf0cada1d77c459a0c8e5973179f1cfd5b0f1ab789d45b17b6440" +dependencies = [ + "protobuf", + "protobuf-codegen", +] + +[[package]] +name = "protocols" +version = "0.1.0" +dependencies = [ + "async-trait", + "protobuf", + "serde", + "serde_json", + "ttrpc", + "ttrpc-codegen", +] + +[[package]] +name = "quote" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1feb54ed693b93a84e14094943b84b7c4eae204c512b7ccb95ab0c66d278ad1" +dependencies = [ + "proc-macro2", +] + [[package]] name = "redox_syscall" version = "0.2.10" @@ -167,6 +589,20 @@ name = "serde" version = "1.0.133" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97565067517b60e2d1ea8b268e59ce036de907ac523ad83a0475da04e818989a" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.133" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed201699328568d8d08208fdd080e3ff594e6c422e438b6705905da01005d537" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] [[package]] name = "serde_json" @@ -179,6 +615,12 @@ dependencies = [ "serde", ] +[[package]] +name = "slab" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb703cfe953bccee95685111adeedb76fabe4e97549a58d16f03ea7b9367bb32" + [[package]] name = "slog" version = "2.7.0" @@ -220,6 +662,27 @@ dependencies = [ "slog", ] +[[package]] +name = "socket2" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66d72b759436ae32898a2af0a14218dbf55efde3feeb170eb623637db85ee1e0" +dependencies = [ + "libc", + "winapi", +] + +[[package]] +name = "syn" +version = "1.0.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b683b2b825c8eef438b77c36a06dc262294da3d5a5813fac20da149241dcd44d" +dependencies = [ + "proc-macro2", + "quote", + "unicode-xid", +] + [[package]] name = "take_mut" version = "0.2.2" @@ -240,6 +703,26 @@ dependencies = [ "winapi", ] +[[package]] +name = "thiserror" +version = "1.0.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "854babe52e4df1653706b98fcfc05843010039b406875930a70e4d9644e5c417" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aa32fd3f627f367fe16f893e2597ae3c05020f8bba2666a4e6ea73d377e5714b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "thread_local" version = "1.1.3" @@ -256,16 +739,141 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6db9e6914ab8b1ae1c260a4ae7a49b6c5611b40328a735b21862567685e73255" dependencies = [ "libc", - "wasi", + "wasi 0.10.0+wasi-snapshot-preview1", "winapi", ] +[[package]] +name = "tokio" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af73ac49756f3f7c01172e34a23e5d0216f6c32333757c2c61feb2bbff5a5ee" +dependencies = [ + "bytes 1.1.0", + "libc", + "memchr", + "mio", + "pin-project-lite", + "socket2", + "tokio-macros", + "winapi", +] + +[[package]] +name = "tokio-macros" +version = "1.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b557f72f448c511a979e2564e55d74e6c4432fc96ff4f6241bc6bded342643b7" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "tokio-vsock" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e0723fc001950a3b018947b05eeb45014fd2b7c6e8f292502193ab74486bdb6" +dependencies = [ + "bytes 0.4.12", + "futures", + "libc", + "tokio", + "vsock", +] + +[[package]] +name = "ttrpc" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66a973ce6d5eaa20c173635b29ffb660dafbc7ef109172c0015ba44e47a23711" +dependencies = [ + "async-trait", + "byteorder", + "futures", + "libc", + "log", + "nix 0.20.2", + "protobuf", + "protobuf-codegen-pure", + "thiserror", + "tokio", + "tokio-vsock", +] + +[[package]] +name = "ttrpc-codegen" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "809eda4e459820237104e4b61d6b41bbe6c9e1ce6adf4057955e6e6722a90408" +dependencies = [ + "protobuf", + "protobuf-codegen", + "protobuf-codegen-pure", + "ttrpc-compiler", +] + +[[package]] +name = "ttrpc-compiler" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2978ed3fa047d8fd55cbeb4d4a61d461fb3021a90c9618519c73ce7e5bb66c15" +dependencies = [ + "derive-new", + "prost", + "prost-build", + "prost-types", + "protobuf", + "protobuf-codegen", + "tempfile", +] + +[[package]] +name = "unicode-segmentation" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e8820f5d777f6224dc4be3632222971ac30164d4a258d595640799554ebfd99" + +[[package]] +name = "unicode-xid" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3" + +[[package]] +name = "vsock" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e32675ee2b3ce5df274c0ab52d19b28789632406277ca26bffee79a8e27dc133" +dependencies = [ + "libc", + "nix 0.23.1", +] + [[package]] name = "wasi" version = "0.10.0+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1a143597ca7c7793eff794def352d41792a93c481eb1042423ff7ff72ba2c31f" +[[package]] +name = "wasi" +version = "0.11.0+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" + +[[package]] +name = "which" +version = "4.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c4fb54e6113b6a8772ee41c3404fb0301ac79604489467e0a9ce1f3e97c24ae" +dependencies = [ + "either", + "lazy_static", + "libc", +] + [[package]] name = "winapi" version = "0.3.9" diff --git a/src/libs/Cargo.toml b/src/libs/Cargo.toml index f4b707a14..16eedb91f 100644 --- a/src/libs/Cargo.toml +++ b/src/libs/Cargo.toml @@ -2,5 +2,6 @@ members = [ "logging", "safe-path", + "protocols", ] resolver = "2" diff --git a/src/libs/protocols/protos/agent.proto b/src/libs/protocols/protos/agent.proto index 13a9094a8..504cae83a 100644 --- a/src/libs/protocols/protos/agent.proto +++ b/src/libs/protocols/protos/agent.proto @@ -51,6 +51,8 @@ service AgentService { rpc ListInterfaces(ListInterfacesRequest) returns(Interfaces); rpc ListRoutes(ListRoutesRequest) returns (Routes); rpc AddARPNeighbors(AddARPNeighborsRequest) returns (google.protobuf.Empty); + rpc GetIPTables(GetIPTablesRequest) returns (GetIPTablesResponse); + rpc SetIPTables(SetIPTablesRequest) returns (SetIPTablesResponse); // observability rpc GetMetrics(GetMetricsRequest) returns (Metrics); @@ -328,6 +330,28 @@ message AddARPNeighborsRequest { ARPNeighbors neighbors = 1; } +message GetIPTablesRequest { + bool is_ipv6 = 1; +} + +message GetIPTablesResponse{ + // raw stdout from iptables-save or ip6tables-save + bytes data = 1; +} + +message SetIPTablesRequest { + bool is_ipv6 = 1; + + // iptables, in raw format expected to be passed to stdin + // of iptables-save or ip6tables-save + bytes data = 2; +} + +message SetIPTablesResponse{ + // raw stdout from iptables-restore or ip6tables-restore + bytes data = 1; +} + message OnlineCPUMemRequest { // Wait specifies if the caller waits for the agent to online all resources. // If true the agent returns once all resources have been connected, otherwise all From 03176a9e09c8835dc03657bcb80a42e232a26377 Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Thu, 21 Apr 2022 21:19:18 -0700 Subject: [PATCH 11/35] proto: update generated code based on proto update Update the generated agent.pb.go code based on proto update. Signed-off-by: Eric Ernst --- .../pkg/agent/protocols/grpc/agent.pb.go | 1240 ++++++++++++++--- 1 file changed, 1024 insertions(+), 216 deletions(-) diff --git a/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go b/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go index 3a3e78504..64b19aa3b 100644 --- a/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go +++ b/src/runtime/virtcontainers/pkg/agent/protocols/grpc/agent.pb.go @@ -1681,6 +1681,167 @@ func (m *AddARPNeighborsRequest) XXX_DiscardUnknown() { var xxx_messageInfo_AddARPNeighborsRequest proto.InternalMessageInfo +type GetIPTablesRequest struct { + IsIpv6 bool `protobuf:"varint,1,opt,name=is_ipv6,json=isIpv6,proto3" json:"is_ipv6,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *GetIPTablesRequest) Reset() { *m = GetIPTablesRequest{} } +func (*GetIPTablesRequest) ProtoMessage() {} +func (*GetIPTablesRequest) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{40} +} +func (m *GetIPTablesRequest) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *GetIPTablesRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_GetIPTablesRequest.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *GetIPTablesRequest) XXX_Merge(src proto.Message) { + xxx_messageInfo_GetIPTablesRequest.Merge(m, src) +} +func (m *GetIPTablesRequest) XXX_Size() int { + return m.Size() +} +func (m *GetIPTablesRequest) XXX_DiscardUnknown() { + xxx_messageInfo_GetIPTablesRequest.DiscardUnknown(m) +} + +var xxx_messageInfo_GetIPTablesRequest proto.InternalMessageInfo + +type GetIPTablesResponse struct { + // raw stdout from iptables-save or ip6tables-save + Data []byte `protobuf:"bytes,1,opt,name=data,proto3" json:"data,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *GetIPTablesResponse) Reset() { *m = GetIPTablesResponse{} } +func (*GetIPTablesResponse) ProtoMessage() {} +func (*GetIPTablesResponse) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{41} +} +func (m *GetIPTablesResponse) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *GetIPTablesResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_GetIPTablesResponse.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *GetIPTablesResponse) XXX_Merge(src proto.Message) { + xxx_messageInfo_GetIPTablesResponse.Merge(m, src) +} +func (m *GetIPTablesResponse) XXX_Size() int { + return m.Size() +} +func (m *GetIPTablesResponse) XXX_DiscardUnknown() { + xxx_messageInfo_GetIPTablesResponse.DiscardUnknown(m) +} + +var xxx_messageInfo_GetIPTablesResponse proto.InternalMessageInfo + +type SetIPTablesRequest struct { + IsIpv6 bool `protobuf:"varint,1,opt,name=is_ipv6,json=isIpv6,proto3" json:"is_ipv6,omitempty"` + // iptables, in raw format expected to be passed to stdin + // of iptables-save or ip6tables-save + Data []byte `protobuf:"bytes,2,opt,name=data,proto3" json:"data,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *SetIPTablesRequest) Reset() { *m = SetIPTablesRequest{} } +func (*SetIPTablesRequest) ProtoMessage() {} +func (*SetIPTablesRequest) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{42} +} +func (m *SetIPTablesRequest) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *SetIPTablesRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_SetIPTablesRequest.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *SetIPTablesRequest) XXX_Merge(src proto.Message) { + xxx_messageInfo_SetIPTablesRequest.Merge(m, src) +} +func (m *SetIPTablesRequest) XXX_Size() int { + return m.Size() +} +func (m *SetIPTablesRequest) XXX_DiscardUnknown() { + xxx_messageInfo_SetIPTablesRequest.DiscardUnknown(m) +} + +var xxx_messageInfo_SetIPTablesRequest proto.InternalMessageInfo + +type SetIPTablesResponse struct { + // raw stdout from iptables-restore or ip6tables-restore ? + Data []byte `protobuf:"bytes,1,opt,name=data,proto3" json:"data,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *SetIPTablesResponse) Reset() { *m = SetIPTablesResponse{} } +func (*SetIPTablesResponse) ProtoMessage() {} +func (*SetIPTablesResponse) Descriptor() ([]byte, []int) { + return fileDescriptor_712ce9a559fda969, []int{43} +} +func (m *SetIPTablesResponse) XXX_Unmarshal(b []byte) error { + return m.Unmarshal(b) +} +func (m *SetIPTablesResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + if deterministic { + return xxx_messageInfo_SetIPTablesResponse.Marshal(b, m, deterministic) + } else { + b = b[:cap(b)] + n, err := m.MarshalToSizedBuffer(b) + if err != nil { + return nil, err + } + return b[:n], nil + } +} +func (m *SetIPTablesResponse) XXX_Merge(src proto.Message) { + xxx_messageInfo_SetIPTablesResponse.Merge(m, src) +} +func (m *SetIPTablesResponse) XXX_Size() int { + return m.Size() +} +func (m *SetIPTablesResponse) XXX_DiscardUnknown() { + xxx_messageInfo_SetIPTablesResponse.DiscardUnknown(m) +} + +var xxx_messageInfo_SetIPTablesResponse proto.InternalMessageInfo + type OnlineCPUMemRequest struct { // Wait specifies if the caller waits for the agent to online all resources. // If true the agent returns once all resources have been connected, otherwise all @@ -1698,7 +1859,7 @@ type OnlineCPUMemRequest struct { func (m *OnlineCPUMemRequest) Reset() { *m = OnlineCPUMemRequest{} } func (*OnlineCPUMemRequest) ProtoMessage() {} func (*OnlineCPUMemRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{40} + return fileDescriptor_712ce9a559fda969, []int{44} } func (m *OnlineCPUMemRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1738,7 +1899,7 @@ type ReseedRandomDevRequest struct { func (m *ReseedRandomDevRequest) Reset() { *m = ReseedRandomDevRequest{} } func (*ReseedRandomDevRequest) ProtoMessage() {} func (*ReseedRandomDevRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{41} + return fileDescriptor_712ce9a559fda969, []int{45} } func (m *ReseedRandomDevRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1788,7 +1949,7 @@ type AgentDetails struct { func (m *AgentDetails) Reset() { *m = AgentDetails{} } func (*AgentDetails) ProtoMessage() {} func (*AgentDetails) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{42} + return fileDescriptor_712ce9a559fda969, []int{46} } func (m *AgentDetails) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1834,7 +1995,7 @@ type GuestDetailsRequest struct { func (m *GuestDetailsRequest) Reset() { *m = GuestDetailsRequest{} } func (*GuestDetailsRequest) ProtoMessage() {} func (*GuestDetailsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{43} + return fileDescriptor_712ce9a559fda969, []int{47} } func (m *GuestDetailsRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1876,7 +2037,7 @@ type GuestDetailsResponse struct { func (m *GuestDetailsResponse) Reset() { *m = GuestDetailsResponse{} } func (*GuestDetailsResponse) ProtoMessage() {} func (*GuestDetailsResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{44} + return fileDescriptor_712ce9a559fda969, []int{48} } func (m *GuestDetailsResponse) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1917,7 +2078,7 @@ type MemHotplugByProbeRequest struct { func (m *MemHotplugByProbeRequest) Reset() { *m = MemHotplugByProbeRequest{} } func (*MemHotplugByProbeRequest) ProtoMessage() {} func (*MemHotplugByProbeRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{45} + return fileDescriptor_712ce9a559fda969, []int{49} } func (m *MemHotplugByProbeRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -1959,7 +2120,7 @@ type SetGuestDateTimeRequest struct { func (m *SetGuestDateTimeRequest) Reset() { *m = SetGuestDateTimeRequest{} } func (*SetGuestDateTimeRequest) ProtoMessage() {} func (*SetGuestDateTimeRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{46} + return fileDescriptor_712ce9a559fda969, []int{50} } func (m *SetGuestDateTimeRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2005,7 +2166,7 @@ type FSGroup struct { func (m *FSGroup) Reset() { *m = FSGroup{} } func (*FSGroup) ProtoMessage() {} func (*FSGroup) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{47} + return fileDescriptor_712ce9a559fda969, []int{51} } func (m *FSGroup) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2073,7 +2234,7 @@ type Storage struct { func (m *Storage) Reset() { *m = Storage{} } func (*Storage) ProtoMessage() {} func (*Storage) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{48} + return fileDescriptor_712ce9a559fda969, []int{52} } func (m *Storage) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2144,7 +2305,7 @@ type Device struct { func (m *Device) Reset() { *m = Device{} } func (*Device) ProtoMessage() {} func (*Device) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{49} + return fileDescriptor_712ce9a559fda969, []int{53} } func (m *Device) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2185,7 +2346,7 @@ type StringUser struct { func (m *StringUser) Reset() { *m = StringUser{} } func (*StringUser) ProtoMessage() {} func (*StringUser) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{50} + return fileDescriptor_712ce9a559fda969, []int{54} } func (m *StringUser) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2242,7 +2403,7 @@ type CopyFileRequest struct { func (m *CopyFileRequest) Reset() { *m = CopyFileRequest{} } func (*CopyFileRequest) ProtoMessage() {} func (*CopyFileRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{51} + return fileDescriptor_712ce9a559fda969, []int{55} } func (m *CopyFileRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2280,7 +2441,7 @@ type GetOOMEventRequest struct { func (m *GetOOMEventRequest) Reset() { *m = GetOOMEventRequest{} } func (*GetOOMEventRequest) ProtoMessage() {} func (*GetOOMEventRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{52} + return fileDescriptor_712ce9a559fda969, []int{56} } func (m *GetOOMEventRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2319,7 +2480,7 @@ type OOMEvent struct { func (m *OOMEvent) Reset() { *m = OOMEvent{} } func (*OOMEvent) ProtoMessage() {} func (*OOMEvent) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{53} + return fileDescriptor_712ce9a559fda969, []int{57} } func (m *OOMEvent) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2358,7 +2519,7 @@ type AddSwapRequest struct { func (m *AddSwapRequest) Reset() { *m = AddSwapRequest{} } func (*AddSwapRequest) ProtoMessage() {} func (*AddSwapRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{54} + return fileDescriptor_712ce9a559fda969, []int{58} } func (m *AddSwapRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2396,7 +2557,7 @@ type GetMetricsRequest struct { func (m *GetMetricsRequest) Reset() { *m = GetMetricsRequest{} } func (*GetMetricsRequest) ProtoMessage() {} func (*GetMetricsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{55} + return fileDescriptor_712ce9a559fda969, []int{59} } func (m *GetMetricsRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2435,7 +2596,7 @@ type Metrics struct { func (m *Metrics) Reset() { *m = Metrics{} } func (*Metrics) ProtoMessage() {} func (*Metrics) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{56} + return fileDescriptor_712ce9a559fda969, []int{60} } func (m *Metrics) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2475,7 +2636,7 @@ type VolumeStatsRequest struct { func (m *VolumeStatsRequest) Reset() { *m = VolumeStatsRequest{} } func (*VolumeStatsRequest) ProtoMessage() {} func (*VolumeStatsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{57} + return fileDescriptor_712ce9a559fda969, []int{61} } func (m *VolumeStatsRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2516,7 +2677,7 @@ type ResizeVolumeRequest struct { func (m *ResizeVolumeRequest) Reset() { *m = ResizeVolumeRequest{} } func (*ResizeVolumeRequest) ProtoMessage() {} func (*ResizeVolumeRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_712ce9a559fda969, []int{58} + return fileDescriptor_712ce9a559fda969, []int{62} } func (m *ResizeVolumeRequest) XXX_Unmarshal(b []byte) error { return m.Unmarshal(b) @@ -2588,6 +2749,10 @@ func init() { proto.RegisterType((*ListRoutesRequest)(nil), "grpc.ListRoutesRequest") proto.RegisterType((*ARPNeighbors)(nil), "grpc.ARPNeighbors") proto.RegisterType((*AddARPNeighborsRequest)(nil), "grpc.AddARPNeighborsRequest") + proto.RegisterType((*GetIPTablesRequest)(nil), "grpc.GetIPTablesRequest") + proto.RegisterType((*GetIPTablesResponse)(nil), "grpc.GetIPTablesResponse") + proto.RegisterType((*SetIPTablesRequest)(nil), "grpc.SetIPTablesRequest") + proto.RegisterType((*SetIPTablesResponse)(nil), "grpc.SetIPTablesResponse") proto.RegisterType((*OnlineCPUMemRequest)(nil), "grpc.OnlineCPUMemRequest") proto.RegisterType((*ReseedRandomDevRequest)(nil), "grpc.ReseedRandomDevRequest") proto.RegisterType((*AgentDetails)(nil), "grpc.AgentDetails") @@ -2614,203 +2779,208 @@ func init() { } var fileDescriptor_712ce9a559fda969 = []byte{ - // 3127 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x1a, 0xcb, 0x72, 0x24, 0x47, - 0xd1, 0xf3, 0x90, 0x66, 0x26, 0xe7, 0xa5, 0x69, 0x69, 0xb5, 0xb3, 0x63, 0x5b, 0xac, 0x7b, 0xed, - 0xb5, 0x6c, 0x63, 0xc9, 0x5e, 0x3b, 0x58, 0x3f, 0xc2, 0x2c, 0x92, 0x56, 0x96, 0x64, 0x5b, 0xde, - 0xa1, 0x65, 0x61, 0x02, 0x02, 0x3a, 0x7a, 0xba, 0x4b, 0x33, 0x65, 0x4d, 0x77, 0xb5, 0xab, 0xab, - 0xb5, 0x1a, 0x13, 0x41, 0x70, 0x82, 0x1b, 0x47, 0x6e, 0xfc, 0x00, 0xc1, 0x1f, 0x70, 0xe1, 0xc0, - 0xc1, 0xc1, 0x89, 0x23, 0x17, 0x08, 0xbc, 0x9f, 0xc0, 0x17, 0x10, 0xf5, 0xea, 0xc7, 0x3c, 0x64, - 0x50, 0x6c, 0x04, 0x97, 0x89, 0xce, 0xac, 0xac, 0x7c, 0x55, 0x65, 0x56, 0x66, 0xd5, 0x40, 0x7f, - 0x88, 0xd9, 0x28, 0x1e, 0x6c, 0xb9, 0xc4, 0xdf, 0x3e, 0x77, 0x98, 0xf3, 0xba, 0x4b, 0x02, 0xe6, - 0xe0, 0x00, 0xd1, 0x68, 0x06, 0x8e, 0xa8, 0xbb, 0x3d, 0xc6, 0x83, 0x68, 0x3b, 0xa4, 0x84, 0x11, - 0x97, 0x8c, 0xd5, 0x57, 0xb4, 0xed, 0x0c, 0x51, 0xc0, 0xb6, 0x04, 0x60, 0x94, 0x87, 0x34, 0x74, - 0x7b, 0x35, 0xe2, 0x62, 0x89, 0xe8, 0xd5, 0xdc, 0x48, 0x7f, 0xd6, 0xd9, 0x24, 0x44, 0x91, 0x02, - 0x9e, 0x1d, 0x12, 0x32, 0x1c, 0x23, 0xc9, 0x63, 0x10, 0x9f, 0x6d, 0x23, 0x3f, 0x64, 0x13, 0x39, - 0x68, 0xfe, 0xbe, 0x08, 0xeb, 0x7b, 0x14, 0x39, 0x0c, 0xed, 0x69, 0x05, 0x2c, 0xf4, 0x65, 0x8c, - 0x22, 0x66, 0xbc, 0x00, 0x8d, 0x44, 0x29, 0x1b, 0x7b, 0xdd, 0xc2, 0xed, 0xc2, 0x66, 0xcd, 0xaa, - 0x27, 0xb8, 0x23, 0xcf, 0xb8, 0x09, 0x15, 0x74, 0x89, 0x5c, 0x3e, 0x5a, 0x14, 0xa3, 0xcb, 0x1c, - 0x3c, 0xf2, 0x8c, 0x37, 0xa1, 0x1e, 0x31, 0x8a, 0x83, 0xa1, 0x1d, 0x47, 0x88, 0x76, 0x4b, 0xb7, - 0x0b, 0x9b, 0xf5, 0x7b, 0x2b, 0x5b, 0x5c, 0xe5, 0xad, 0x13, 0x31, 0x70, 0x1a, 0x21, 0x6a, 0x41, - 0x94, 0x7c, 0x1b, 0x77, 0xa1, 0xe2, 0xa1, 0x0b, 0xec, 0xa2, 0xa8, 0x5b, 0xbe, 0x5d, 0xda, 0xac, - 0xdf, 0x6b, 0x48, 0xf2, 0x87, 0x02, 0x69, 0xe9, 0x41, 0xe3, 0x15, 0xa8, 0x46, 0x8c, 0x50, 0x67, - 0x88, 0xa2, 0xee, 0x92, 0x20, 0x6c, 0x6a, 0xbe, 0x02, 0x6b, 0x25, 0xc3, 0xc6, 0x73, 0x50, 0x7a, - 0xb4, 0x77, 0xd4, 0x5d, 0x16, 0xd2, 0x41, 0x51, 0x85, 0xc8, 0xb5, 0x38, 0xda, 0xb8, 0x03, 0xcd, - 0xc8, 0x09, 0xbc, 0x01, 0xb9, 0xb4, 0x43, 0xec, 0x05, 0x51, 0xb7, 0x72, 0xbb, 0xb0, 0x59, 0xb5, - 0x1a, 0x0a, 0xd9, 0xe7, 0x38, 0xf3, 0x3d, 0xb8, 0x71, 0xc2, 0x1c, 0xca, 0xae, 0xe1, 0x1d, 0xf3, - 0x14, 0xd6, 0x2d, 0xe4, 0x93, 0x8b, 0x6b, 0xb9, 0xb6, 0x0b, 0x15, 0x86, 0x7d, 0x44, 0x62, 0x26, - 0x5c, 0xdb, 0xb4, 0x34, 0x68, 0xfe, 0xb1, 0x00, 0xc6, 0xfe, 0x25, 0x72, 0xfb, 0x94, 0xb8, 0x28, - 0x8a, 0xfe, 0x4f, 0xcb, 0xf5, 0x32, 0x54, 0x42, 0xa9, 0x40, 0xb7, 0x2c, 0xc8, 0xd5, 0x2a, 0x68, - 0xad, 0xf4, 0xa8, 0xf9, 0x05, 0xac, 0x9d, 0xe0, 0x61, 0xe0, 0x8c, 0x9f, 0xa2, 0xbe, 0xeb, 0xb0, - 0x1c, 0x09, 0x9e, 0x42, 0xd5, 0xa6, 0xa5, 0x20, 0xb3, 0x0f, 0xc6, 0xe7, 0x0e, 0x66, 0x4f, 0x4f, - 0x92, 0xf9, 0x3a, 0xac, 0xe6, 0x38, 0x46, 0x21, 0x09, 0x22, 0x24, 0x14, 0x60, 0x0e, 0x8b, 0x23, - 0xc1, 0x6c, 0xc9, 0x52, 0x90, 0x49, 0x60, 0xfd, 0x34, 0xf4, 0xae, 0x19, 0x4d, 0xf7, 0xa0, 0x46, - 0x51, 0x44, 0x62, 0xca, 0x63, 0xa0, 0x28, 0x9c, 0xba, 0x26, 0x9d, 0xfa, 0x09, 0x0e, 0xe2, 0x4b, - 0x4b, 0x8f, 0x59, 0x29, 0x99, 0xda, 0x9f, 0x2c, 0xba, 0xce, 0xfe, 0x7c, 0x0f, 0x6e, 0xf4, 0x9d, - 0x38, 0xba, 0x8e, 0xae, 0xe6, 0xfb, 0x7c, 0x6f, 0x47, 0xb1, 0x7f, 0xad, 0xc9, 0x7f, 0x28, 0x40, - 0x75, 0x2f, 0x8c, 0x4f, 0x23, 0x67, 0x88, 0x8c, 0xef, 0x40, 0x9d, 0x11, 0xe6, 0x8c, 0xed, 0x98, - 0x83, 0x82, 0xbc, 0x6c, 0x81, 0x40, 0x49, 0x82, 0x17, 0xa0, 0x11, 0x22, 0xea, 0x86, 0xb1, 0xa2, - 0x28, 0xde, 0x2e, 0x6d, 0x96, 0xad, 0xba, 0xc4, 0x49, 0x92, 0x2d, 0x58, 0x15, 0x63, 0x36, 0x0e, - 0xec, 0x73, 0x44, 0x03, 0x34, 0xf6, 0x89, 0x87, 0xc4, 0xe6, 0x28, 0x5b, 0x1d, 0x31, 0x74, 0x14, - 0x7c, 0x9c, 0x0c, 0x18, 0xaf, 0x42, 0x27, 0xa1, 0xe7, 0x3b, 0x5e, 0x50, 0x97, 0x05, 0x75, 0x5b, - 0x51, 0x9f, 0x2a, 0xb4, 0xf9, 0x4b, 0x68, 0x7d, 0x36, 0xa2, 0x84, 0xb1, 0x31, 0x0e, 0x86, 0x0f, - 0x1d, 0xe6, 0xf0, 0xd0, 0x0c, 0x11, 0xc5, 0xc4, 0x8b, 0x94, 0xb6, 0x1a, 0x34, 0x5e, 0x83, 0x0e, - 0x93, 0xb4, 0xc8, 0xb3, 0x35, 0x4d, 0x51, 0xd0, 0xac, 0x24, 0x03, 0x7d, 0x45, 0xfc, 0x12, 0xb4, - 0x52, 0x62, 0x1e, 0xdc, 0x4a, 0xdf, 0x66, 0x82, 0xfd, 0x0c, 0xfb, 0xc8, 0xbc, 0x10, 0xbe, 0x12, - 0x8b, 0x6c, 0xbc, 0x06, 0xb5, 0xd4, 0x0f, 0x05, 0xb1, 0x43, 0x5a, 0x72, 0x87, 0x68, 0x77, 0x5a, - 0xd5, 0xc4, 0x29, 0x1f, 0x40, 0x9b, 0x25, 0x8a, 0xdb, 0x9e, 0xc3, 0x9c, 0xfc, 0xa6, 0xca, 0x5b, - 0x65, 0xb5, 0x58, 0x0e, 0x36, 0xdf, 0x87, 0x5a, 0x1f, 0x7b, 0x91, 0x14, 0xdc, 0x85, 0x8a, 0x1b, - 0x53, 0x8a, 0x02, 0xa6, 0x4d, 0x56, 0xa0, 0xb1, 0x06, 0x4b, 0x63, 0xec, 0x63, 0xa6, 0xcc, 0x94, - 0x80, 0x49, 0x00, 0x8e, 0x91, 0x4f, 0xe8, 0x44, 0x38, 0x6c, 0x0d, 0x96, 0xb2, 0x8b, 0x2b, 0x01, - 0xe3, 0x59, 0xa8, 0xf9, 0xce, 0x65, 0xb2, 0xa8, 0x7c, 0xa4, 0xea, 0x3b, 0x97, 0x52, 0xf9, 0x2e, - 0x54, 0xce, 0x1c, 0x3c, 0x76, 0x03, 0xa6, 0xbc, 0xa2, 0xc1, 0x54, 0x60, 0x39, 0x2b, 0xf0, 0x2f, - 0x45, 0xa8, 0x4b, 0x89, 0x52, 0xe1, 0x35, 0x58, 0x72, 0x1d, 0x77, 0x94, 0x88, 0x14, 0x80, 0x71, - 0x57, 0x2b, 0x52, 0xcc, 0x66, 0xb8, 0x54, 0x53, 0xad, 0xda, 0x36, 0x40, 0xf4, 0xd8, 0x09, 0x95, - 0x6e, 0xa5, 0x05, 0xc4, 0x35, 0x4e, 0x23, 0xd5, 0x7d, 0x0b, 0x1a, 0x72, 0xdf, 0xa9, 0x29, 0xe5, - 0x05, 0x53, 0xea, 0x92, 0x4a, 0x4e, 0xba, 0x03, 0xcd, 0x38, 0x42, 0xf6, 0x08, 0x23, 0xea, 0x50, - 0x77, 0x34, 0xe9, 0x2e, 0xc9, 0x03, 0x28, 0x8e, 0xd0, 0xa1, 0xc6, 0x19, 0xf7, 0x60, 0x89, 0xe7, - 0x96, 0xa8, 0xbb, 0x2c, 0xce, 0xba, 0xe7, 0xb2, 0x2c, 0x85, 0xa9, 0x5b, 0xe2, 0x77, 0x3f, 0x60, - 0x74, 0x62, 0x49, 0xd2, 0xde, 0x3b, 0x00, 0x29, 0xd2, 0x58, 0x81, 0xd2, 0x39, 0x9a, 0xa8, 0x38, - 0xe4, 0x9f, 0xdc, 0x39, 0x17, 0xce, 0x38, 0xd6, 0x5e, 0x97, 0xc0, 0x7b, 0xc5, 0x77, 0x0a, 0xa6, - 0x0b, 0xed, 0xdd, 0xf1, 0x39, 0x26, 0x99, 0xe9, 0x6b, 0xb0, 0xe4, 0x3b, 0x5f, 0x10, 0xaa, 0x3d, - 0x29, 0x00, 0x81, 0xc5, 0x01, 0xa1, 0x9a, 0x85, 0x00, 0x8c, 0x16, 0x14, 0x49, 0x28, 0xfc, 0x55, - 0xb3, 0x8a, 0x24, 0x4c, 0x05, 0x95, 0x33, 0x82, 0xcc, 0x7f, 0x96, 0x01, 0x52, 0x29, 0x86, 0x05, - 0x3d, 0x4c, 0xec, 0x08, 0x51, 0x7e, 0xbe, 0xdb, 0x83, 0x09, 0x43, 0x91, 0x4d, 0x91, 0x1b, 0xd3, - 0x08, 0x5f, 0xf0, 0xf5, 0xe3, 0x66, 0xdf, 0x90, 0x66, 0x4f, 0xe9, 0x66, 0xdd, 0xc4, 0xe4, 0x44, - 0xce, 0xdb, 0xe5, 0xd3, 0x2c, 0x3d, 0xcb, 0x38, 0x82, 0x1b, 0x29, 0x4f, 0x2f, 0xc3, 0xae, 0x78, - 0x15, 0xbb, 0xd5, 0x84, 0x9d, 0x97, 0xb2, 0xda, 0x87, 0x55, 0x4c, 0xec, 0x2f, 0x63, 0x14, 0xe7, - 0x18, 0x95, 0xae, 0x62, 0xd4, 0xc1, 0xe4, 0x87, 0x62, 0x42, 0xca, 0xa6, 0x0f, 0xb7, 0x32, 0x56, - 0xf2, 0x70, 0xcf, 0x30, 0x2b, 0x5f, 0xc5, 0x6c, 0x3d, 0xd1, 0x8a, 0xe7, 0x83, 0x94, 0xe3, 0x47, - 0xb0, 0x8e, 0x89, 0xfd, 0xd8, 0xc1, 0x6c, 0x9a, 0xdd, 0xd2, 0xb7, 0x18, 0xc9, 0x4f, 0xb4, 0x3c, - 0x2f, 0x69, 0xa4, 0x8f, 0xe8, 0x30, 0x67, 0xe4, 0xf2, 0xb7, 0x18, 0x79, 0x2c, 0x26, 0xa4, 0x6c, - 0x76, 0xa0, 0x83, 0xc9, 0xb4, 0x36, 0x95, 0xab, 0x98, 0xb4, 0x31, 0xc9, 0x6b, 0xb2, 0x0b, 0x9d, - 0x08, 0xb9, 0x8c, 0xd0, 0xec, 0x26, 0xa8, 0x5e, 0xc5, 0x62, 0x45, 0xd1, 0x27, 0x3c, 0xcc, 0x9f, - 0x42, 0xe3, 0x30, 0x1e, 0x22, 0x36, 0x1e, 0x24, 0xc9, 0xe0, 0xa9, 0xe5, 0x1f, 0xf3, 0xdf, 0x45, - 0xa8, 0xef, 0x0d, 0x29, 0x89, 0xc3, 0x5c, 0x4e, 0x96, 0x41, 0x3a, 0x9d, 0x93, 0x05, 0x89, 0xc8, - 0xc9, 0x92, 0xf8, 0x6d, 0x68, 0xf8, 0x22, 0x74, 0x15, 0xbd, 0xcc, 0x43, 0x9d, 0x99, 0xa0, 0xb6, - 0xea, 0x7e, 0x26, 0x99, 0x6d, 0x01, 0x84, 0xd8, 0x8b, 0xd4, 0x1c, 0x99, 0x8e, 0xda, 0xaa, 0xdc, - 0xd2, 0x29, 0xda, 0xaa, 0x85, 0x49, 0xb6, 0x7e, 0x13, 0xea, 0x03, 0xee, 0x24, 0x35, 0x21, 0x97, - 0x8c, 0x52, 0xef, 0x59, 0x30, 0x48, 0x83, 0xf0, 0x10, 0x9a, 0x23, 0xe9, 0x32, 0x35, 0x49, 0xee, - 0xa1, 0x3b, 0xca, 0x92, 0xd4, 0xde, 0xad, 0xac, 0x67, 0xe5, 0x02, 0x34, 0x46, 0x19, 0x54, 0xef, - 0x04, 0x3a, 0x33, 0x24, 0x73, 0x72, 0xd0, 0x66, 0x36, 0x07, 0xd5, 0xef, 0x19, 0x52, 0x50, 0x76, - 0x66, 0x36, 0x2f, 0xfd, 0xb6, 0x08, 0x8d, 0x4f, 0x11, 0x7b, 0x4c, 0xe8, 0xb9, 0xd4, 0xd7, 0x80, - 0x72, 0xe0, 0xf8, 0x48, 0x71, 0x14, 0xdf, 0xc6, 0x2d, 0xa8, 0xd2, 0x4b, 0x99, 0x40, 0xd4, 0x7a, - 0x56, 0xe8, 0xa5, 0x48, 0x0c, 0xc6, 0xf3, 0x00, 0xf4, 0xd2, 0x0e, 0x1d, 0xf7, 0x1c, 0x29, 0x0f, - 0x96, 0xad, 0x1a, 0xbd, 0xec, 0x4b, 0x04, 0xdf, 0x0a, 0xf4, 0xd2, 0x46, 0x94, 0x12, 0x1a, 0xa9, - 0x5c, 0x55, 0xa5, 0x97, 0xfb, 0x02, 0x56, 0x73, 0x3d, 0x4a, 0xc2, 0x10, 0x79, 0x22, 0x47, 0x8b, - 0xb9, 0x0f, 0x25, 0x82, 0x4b, 0x65, 0x5a, 0xea, 0xb2, 0x94, 0xca, 0x52, 0xa9, 0x2c, 0x95, 0x5a, - 0x91, 0x33, 0x59, 0x56, 0x2a, 0x4b, 0xa4, 0x56, 0xa5, 0x54, 0x96, 0x91, 0xca, 0x52, 0xa9, 0x35, - 0x3d, 0x57, 0x49, 0x35, 0x7f, 0x53, 0x80, 0xf5, 0xe9, 0xc2, 0x4f, 0xd5, 0xa6, 0x6f, 0x43, 0xc3, - 0x15, 0xeb, 0x95, 0xdb, 0x93, 0x9d, 0x99, 0x95, 0xb4, 0xea, 0x6e, 0x66, 0x1b, 0xdf, 0x87, 0x66, - 0x20, 0x1d, 0x9c, 0x6c, 0xcd, 0x52, 0xba, 0x2e, 0x59, 0xdf, 0x5b, 0x8d, 0x20, 0x03, 0x99, 0x1e, - 0x18, 0x9f, 0x53, 0xcc, 0xd0, 0x09, 0xa3, 0xc8, 0xf1, 0x9f, 0x46, 0x75, 0x6f, 0x40, 0x59, 0x54, - 0x2b, 0x7c, 0x99, 0x1a, 0x96, 0xf8, 0x36, 0x5f, 0x86, 0xd5, 0x9c, 0x14, 0x65, 0xeb, 0x0a, 0x94, - 0xc6, 0x28, 0x10, 0xdc, 0x9b, 0x16, 0xff, 0x34, 0x1d, 0xe8, 0x58, 0xc8, 0xf1, 0x9e, 0x9e, 0x36, - 0x4a, 0x44, 0x29, 0x15, 0xb1, 0x09, 0x46, 0x56, 0x84, 0x52, 0x45, 0x6b, 0x5d, 0xc8, 0x68, 0xfd, - 0x08, 0x3a, 0x7b, 0x63, 0x12, 0xa1, 0x13, 0xe6, 0xe1, 0xe0, 0x69, 0xb4, 0x23, 0xbf, 0x80, 0xd5, - 0xcf, 0xd8, 0xe4, 0x73, 0xce, 0x2c, 0xc2, 0x5f, 0xa1, 0xa7, 0x64, 0x1f, 0x25, 0x8f, 0xb5, 0x7d, - 0x94, 0x3c, 0xe6, 0xcd, 0x8d, 0x4b, 0xc6, 0xb1, 0x1f, 0x88, 0x50, 0x68, 0x5a, 0x0a, 0x32, 0x77, - 0xa1, 0x21, 0x6b, 0xe8, 0x63, 0xe2, 0xc5, 0x63, 0x34, 0x37, 0x06, 0x37, 0x00, 0x42, 0x87, 0x3a, - 0x3e, 0x62, 0x88, 0xca, 0x3d, 0x54, 0xb3, 0x32, 0x18, 0xf3, 0x77, 0x45, 0x58, 0x93, 0xf7, 0x0d, - 0x27, 0xb2, 0xcd, 0xd6, 0x26, 0xf4, 0xa0, 0x3a, 0x22, 0x11, 0xcb, 0x30, 0x4c, 0x60, 0xae, 0x22, - 0xef, 0xcf, 0x25, 0x37, 0xfe, 0x99, 0xbb, 0x04, 0x28, 0x5d, 0x7d, 0x09, 0x30, 0xd3, 0xe6, 0x97, - 0x67, 0xdb, 0x7c, 0x1e, 0x6d, 0x9a, 0x08, 0xcb, 0x18, 0xaf, 0x59, 0x35, 0x85, 0x39, 0xf2, 0x8c, - 0xbb, 0xd0, 0x1e, 0x72, 0x2d, 0xed, 0x11, 0x21, 0xe7, 0x76, 0xe8, 0xb0, 0x91, 0x08, 0xf5, 0x9a, - 0xd5, 0x14, 0xe8, 0x43, 0x42, 0xce, 0xfb, 0x0e, 0x1b, 0x19, 0xef, 0x42, 0x4b, 0x95, 0x81, 0xbe, - 0x70, 0x51, 0xa4, 0x0e, 0x3f, 0x15, 0x45, 0x59, 0xef, 0x59, 0xcd, 0xf3, 0x0c, 0x14, 0x99, 0x37, - 0xe1, 0xc6, 0x43, 0x14, 0x31, 0x4a, 0x26, 0x79, 0xc7, 0x98, 0xdf, 0x07, 0x38, 0x0a, 0x18, 0xa2, - 0x67, 0x8e, 0x8b, 0x22, 0xe3, 0x8d, 0x2c, 0xa4, 0x8a, 0xa3, 0x95, 0x2d, 0x79, 0xdd, 0x93, 0x0c, - 0x58, 0x19, 0x1a, 0x73, 0x0b, 0x96, 0x2d, 0x12, 0xf3, 0x74, 0xf4, 0xa2, 0xfe, 0x52, 0xf3, 0x1a, - 0x6a, 0x9e, 0x40, 0x5a, 0x6a, 0xcc, 0x3c, 0xd4, 0x2d, 0x6c, 0xca, 0x4e, 0x2d, 0xd1, 0x16, 0xd4, - 0xb0, 0xc6, 0xa9, 0xac, 0x32, 0x2b, 0x3a, 0x25, 0x31, 0xdf, 0x87, 0x55, 0xc9, 0x49, 0x72, 0xd6, - 0x6c, 0x5e, 0x84, 0x65, 0xaa, 0xd5, 0x28, 0xa4, 0xf7, 0x3c, 0x8a, 0x48, 0x8d, 0x71, 0x7f, 0x7c, - 0x82, 0x23, 0x96, 0x1a, 0xa2, 0xfd, 0xb1, 0x0a, 0x1d, 0x3e, 0x90, 0xe3, 0x69, 0x7e, 0x08, 0x8d, - 0x1d, 0xab, 0xff, 0x29, 0xc2, 0xc3, 0xd1, 0x80, 0x67, 0xcf, 0xef, 0xe5, 0x61, 0x65, 0xb0, 0xa1, - 0xb4, 0xcd, 0x0c, 0x59, 0x39, 0x3a, 0xf3, 0x23, 0x58, 0xdf, 0xf1, 0xbc, 0x2c, 0x4a, 0x6b, 0xfd, - 0x06, 0xd4, 0x82, 0x0c, 0xbb, 0xcc, 0x99, 0x95, 0xa3, 0x4e, 0x89, 0xcc, 0x9f, 0xc1, 0xea, 0xa3, - 0x60, 0x8c, 0x03, 0xb4, 0xd7, 0x3f, 0x3d, 0x46, 0x49, 0x2e, 0x32, 0xa0, 0xcc, 0x6b, 0x36, 0xc1, - 0xa3, 0x6a, 0x89, 0x6f, 0x1e, 0x9c, 0xc1, 0xc0, 0x76, 0xc3, 0x38, 0x52, 0x97, 0x3d, 0xcb, 0xc1, - 0x60, 0x2f, 0x8c, 0x23, 0x7e, 0xb8, 0xf0, 0xe2, 0x82, 0x04, 0xe3, 0x89, 0x88, 0xd0, 0xaa, 0x55, - 0x71, 0xc3, 0xf8, 0x51, 0x30, 0x9e, 0x98, 0xdf, 0x15, 0x1d, 0x38, 0x42, 0x9e, 0xe5, 0x04, 0x1e, - 0xf1, 0x1f, 0xa2, 0x8b, 0x8c, 0x84, 0xa4, 0xdb, 0xd3, 0x99, 0xe8, 0xeb, 0x02, 0x34, 0x76, 0x86, - 0x28, 0x60, 0x0f, 0x11, 0x73, 0xf0, 0x58, 0x74, 0x74, 0x17, 0x88, 0x46, 0x98, 0x04, 0x2a, 0xdc, - 0x34, 0xc8, 0x1b, 0x72, 0x1c, 0x60, 0x66, 0x7b, 0x0e, 0xf2, 0x49, 0x20, 0xb8, 0x54, 0x2d, 0xe0, - 0xa8, 0x87, 0x02, 0x63, 0xbc, 0x0c, 0x6d, 0x79, 0x19, 0x67, 0x8f, 0x9c, 0xc0, 0x1b, 0xf3, 0x40, - 0x2f, 0x89, 0xd0, 0x6c, 0x49, 0xf4, 0xa1, 0xc2, 0x1a, 0xaf, 0xc0, 0x8a, 0x0a, 0xc3, 0x94, 0xb2, - 0x2c, 0x28, 0xdb, 0x0a, 0x9f, 0x23, 0x8d, 0xc3, 0x90, 0x50, 0x16, 0xd9, 0x11, 0x72, 0x5d, 0xe2, - 0x87, 0xaa, 0x1d, 0x6a, 0x6b, 0xfc, 0x89, 0x44, 0x9b, 0x43, 0x58, 0x3d, 0xe0, 0x76, 0x2a, 0x4b, - 0xd2, 0x6d, 0xd5, 0xf2, 0x91, 0x6f, 0x0f, 0xc6, 0xc4, 0x3d, 0xb7, 0x79, 0x72, 0x54, 0x1e, 0xe6, - 0x05, 0xd7, 0x2e, 0x47, 0x9e, 0xe0, 0xaf, 0x44, 0xe7, 0xcf, 0xa9, 0x46, 0x84, 0x85, 0xe3, 0x78, - 0x68, 0x87, 0x94, 0x0c, 0x90, 0x32, 0xb1, 0xed, 0x23, 0xff, 0x50, 0xe2, 0xfb, 0x1c, 0x6d, 0xfe, - 0xa9, 0x00, 0x6b, 0x79, 0x49, 0x2a, 0xd5, 0x6f, 0xc3, 0x5a, 0x5e, 0x94, 0x3a, 0xfe, 0x65, 0x79, - 0xd9, 0xc9, 0x0a, 0x94, 0x85, 0xc0, 0x7d, 0x68, 0x8a, 0xab, 0x5b, 0xdb, 0x93, 0x9c, 0xf2, 0x45, - 0x4f, 0x76, 0x5d, 0xac, 0x86, 0x93, 0x5d, 0xa5, 0x77, 0xe1, 0x96, 0x32, 0xdf, 0x9e, 0x55, 0x5b, - 0x6e, 0x88, 0x75, 0x45, 0x70, 0x3c, 0xa5, 0xfd, 0x27, 0xd0, 0x4d, 0x51, 0xbb, 0x13, 0x81, 0x4c, - 0x37, 0xf3, 0xea, 0x94, 0xb1, 0x3b, 0x9e, 0x47, 0x45, 0x94, 0x94, 0xad, 0x79, 0x43, 0xe6, 0x03, - 0xb8, 0x79, 0x82, 0x98, 0xf4, 0x86, 0xc3, 0x54, 0x27, 0x22, 0x99, 0xad, 0x40, 0xe9, 0x04, 0xb9, - 0xc2, 0xf8, 0x92, 0xc5, 0x3f, 0xf9, 0x06, 0x3c, 0x8d, 0x90, 0x2b, 0xac, 0x2c, 0x59, 0xe2, 0xdb, - 0x0c, 0xa1, 0xf2, 0xe1, 0xc9, 0x01, 0xaf, 0x37, 0xf8, 0xa6, 0x96, 0xf5, 0x89, 0x3a, 0x8b, 0x9a, - 0x56, 0x45, 0xc0, 0x47, 0x9e, 0xf1, 0x11, 0xac, 0xca, 0x21, 0x77, 0xe4, 0x04, 0x43, 0x64, 0x87, - 0x64, 0x8c, 0x5d, 0xb9, 0xf5, 0x5b, 0xf7, 0x7a, 0x2a, 0x7c, 0x15, 0x9f, 0x3d, 0x41, 0xd2, 0x17, - 0x14, 0x56, 0x67, 0x38, 0x8d, 0x32, 0xff, 0x51, 0x80, 0x8a, 0x3a, 0x0e, 0xf8, 0x91, 0xe6, 0x51, - 0x7c, 0x81, 0xa8, 0xda, 0xec, 0x0a, 0x32, 0x5e, 0x82, 0x96, 0xfc, 0xb2, 0x49, 0xc8, 0x30, 0x49, - 0x0e, 0x99, 0xa6, 0xc4, 0x3e, 0x92, 0x48, 0x71, 0xdd, 0x27, 0x2e, 0xdc, 0x54, 0x6f, 0xab, 0x20, - 0x8e, 0x3f, 0x8b, 0xb8, 0x52, 0xe2, 0x50, 0xa9, 0x59, 0x0a, 0xe2, 0xc1, 0xa5, 0xf9, 0x2d, 0x09, - 0x7e, 0x1a, 0xe4, 0xc1, 0xe5, 0x93, 0x38, 0x60, 0x76, 0x48, 0x70, 0xc0, 0xd4, 0x29, 0x02, 0x02, - 0xd5, 0xe7, 0x18, 0x63, 0x13, 0xaa, 0x67, 0x91, 0x2d, 0xac, 0x11, 0x15, 0x63, 0x72, 0xb2, 0x29, - 0xab, 0xad, 0xca, 0x59, 0x24, 0x3e, 0xcc, 0x5f, 0x17, 0x60, 0x59, 0x5e, 0x8e, 0xf3, 0xbe, 0x3b, - 0x39, 0xf5, 0x8b, 0x58, 0x54, 0x50, 0x42, 0x2b, 0x79, 0xd2, 0x8b, 0x6f, 0x9e, 0x63, 0x2e, 0x7c, - 0x79, 0x76, 0x29, 0x23, 0x2e, 0x7c, 0x71, 0x68, 0xbd, 0x04, 0xad, 0xb4, 0x78, 0x10, 0xe3, 0xd2, - 0x98, 0x66, 0x82, 0x15, 0x64, 0x0b, 0x6d, 0x32, 0x7f, 0x0c, 0x90, 0x5e, 0x12, 0xf3, 0xed, 0x10, - 0x27, 0xca, 0xf0, 0x4f, 0x8e, 0x19, 0x26, 0x65, 0x07, 0xff, 0x34, 0xee, 0x42, 0xcb, 0xf1, 0x3c, - 0xcc, 0xa7, 0x3b, 0xe3, 0x03, 0xec, 0x25, 0x09, 0x24, 0x8f, 0x35, 0xff, 0x5a, 0x80, 0xf6, 0x1e, - 0x09, 0x27, 0x1f, 0xe2, 0x31, 0xca, 0x64, 0x37, 0xa1, 0xa4, 0xaa, 0x3a, 0xf8, 0x37, 0xaf, 0xa4, - 0xcf, 0xf0, 0x18, 0xc9, 0xb0, 0x97, 0xbb, 0xae, 0xca, 0x11, 0x22, 0xe4, 0xf5, 0x60, 0x72, 0x25, - 0xd8, 0x94, 0x83, 0xc7, 0xc4, 0x13, 0x3d, 0x83, 0x87, 0xa9, 0x9d, 0x5c, 0x00, 0x36, 0xad, 0x8a, - 0x87, 0xa9, 0x18, 0x52, 0x86, 0x2c, 0x89, 0x0b, 0xde, 0xac, 0x21, 0xcb, 0x12, 0xc3, 0x0d, 0x59, - 0x87, 0x65, 0x72, 0x76, 0x16, 0x21, 0x26, 0xd6, 0xaa, 0x64, 0x29, 0x28, 0x49, 0xc1, 0xd5, 0x4c, - 0x0a, 0x5e, 0x03, 0xe3, 0x00, 0xb1, 0x47, 0x8f, 0x8e, 0xf7, 0x2f, 0x50, 0xc0, 0xf4, 0xc9, 0xf5, - 0x3a, 0x54, 0x35, 0xea, 0xbf, 0xb9, 0x3a, 0x7d, 0x15, 0x5a, 0x3b, 0x9e, 0x77, 0xf2, 0xd8, 0x09, - 0xb5, 0x3f, 0xba, 0x50, 0xe9, 0xef, 0x1d, 0xf5, 0xa5, 0x4b, 0x4a, 0xdc, 0x00, 0x05, 0xf2, 0x93, - 0xf2, 0x00, 0xb1, 0x63, 0xc4, 0x28, 0x76, 0x93, 0x93, 0xf2, 0x0e, 0x54, 0x14, 0x86, 0xcf, 0xf4, - 0xe5, 0xa7, 0x3e, 0x02, 0x14, 0x68, 0xfe, 0x00, 0x8c, 0x1f, 0xf1, 0x9a, 0x0f, 0xc9, 0x82, 0x5f, - 0x49, 0x7a, 0x15, 0x3a, 0x17, 0x02, 0x6b, 0xcb, 0x62, 0x28, 0xb3, 0x0c, 0x6d, 0x39, 0x20, 0xf2, - 0x83, 0x90, 0x7d, 0x0a, 0xab, 0xb2, 0x44, 0x95, 0x7c, 0xae, 0xc1, 0x82, 0xfb, 0x30, 0x59, 0xcf, - 0xb2, 0x25, 0xbe, 0xef, 0xfd, 0xb9, 0xa3, 0x8e, 0x31, 0x75, 0x23, 0x62, 0x1c, 0x40, 0x7b, 0xea, - 0xf9, 0xca, 0x50, 0x57, 0x64, 0xf3, 0x5f, 0xb5, 0x7a, 0xeb, 0x5b, 0xf2, 0x39, 0x6c, 0x4b, 0x3f, - 0x87, 0x6d, 0xed, 0xfb, 0x21, 0x9b, 0x18, 0xfb, 0xd0, 0xca, 0x3f, 0xf4, 0x18, 0xcf, 0xea, 0x8a, - 0x72, 0xce, 0xf3, 0xcf, 0x42, 0x36, 0x07, 0xd0, 0x9e, 0x7a, 0xf3, 0xd1, 0xfa, 0xcc, 0x7f, 0x0a, - 0x5a, 0xc8, 0xe8, 0x01, 0xd4, 0x33, 0x8f, 0x3c, 0x46, 0x57, 0x32, 0x99, 0x7d, 0xf7, 0x59, 0xc8, - 0x60, 0x0f, 0x9a, 0xb9, 0x77, 0x17, 0xa3, 0xa7, 0xec, 0x99, 0xf3, 0x18, 0xb3, 0x90, 0xc9, 0x2e, - 0xd4, 0x33, 0xcf, 0x1f, 0x5a, 0x8b, 0xd9, 0x37, 0x96, 0xde, 0xad, 0x39, 0x23, 0xea, 0xb4, 0x3c, - 0x80, 0xf6, 0xd4, 0x9b, 0x88, 0x76, 0xc9, 0xfc, 0xa7, 0x92, 0x85, 0xca, 0x7c, 0x2c, 0x96, 0x28, - 0xd3, 0xf2, 0x66, 0x96, 0x68, 0xf6, 0x05, 0xa4, 0xf7, 0xdc, 0xfc, 0x41, 0xa5, 0xd5, 0x3e, 0xb4, - 0xf2, 0x8f, 0x1f, 0x9a, 0xd9, 0xdc, 0x27, 0x91, 0xab, 0xd7, 0x3b, 0xf7, 0x0e, 0x92, 0xae, 0xf7, - 0xbc, 0xe7, 0x91, 0x85, 0x8c, 0x76, 0x00, 0x54, 0x83, 0xeb, 0xe1, 0x20, 0x71, 0xf4, 0x4c, 0x63, - 0x9d, 0x38, 0x7a, 0x4e, 0x33, 0xfc, 0x00, 0x40, 0xf6, 0xa5, 0x1e, 0x89, 0x99, 0x71, 0x53, 0xab, - 0x31, 0xd5, 0x0c, 0xf7, 0xba, 0xb3, 0x03, 0x33, 0x0c, 0x10, 0xa5, 0xd7, 0x61, 0xf0, 0x01, 0x40, - 0xda, 0xef, 0x6a, 0x06, 0x33, 0x1d, 0xf0, 0x15, 0x3e, 0x68, 0x64, 0xbb, 0x5b, 0x43, 0xd9, 0x3a, - 0xa7, 0xe3, 0xbd, 0x82, 0x45, 0x7b, 0xaa, 0x7b, 0xc9, 0x6f, 0xb6, 0xe9, 0xa6, 0xa6, 0x37, 0xd3, - 0xc1, 0x18, 0xf7, 0xa1, 0x91, 0x6d, 0x5b, 0xb4, 0x16, 0x73, 0x5a, 0x99, 0x5e, 0xae, 0x75, 0x31, - 0x1e, 0x40, 0x2b, 0xdf, 0xb2, 0xe8, 0x2d, 0x35, 0xb7, 0x91, 0xe9, 0xa9, 0x0b, 0xb9, 0x0c, 0xf9, - 0x5b, 0x00, 0x69, 0x6b, 0xa3, 0xdd, 0x37, 0xd3, 0xec, 0x4c, 0x49, 0x3d, 0x80, 0xf6, 0x54, 0xcb, - 0xa2, 0x2d, 0x9e, 0xdf, 0xc9, 0x2c, 0x74, 0xdd, 0xdb, 0x00, 0xe9, 0x71, 0xa1, 0xa5, 0xcf, 0x1c, - 0x20, 0xbd, 0xa6, 0xbe, 0xac, 0x94, 0x74, 0x7b, 0xd0, 0xcc, 0xf5, 0xf3, 0x3a, 0xcd, 0xcc, 0x6b, - 0xf2, 0xaf, 0x4a, 0xbe, 0xf9, 0xe6, 0x57, 0x7b, 0x6e, 0x6e, 0x4b, 0x7c, 0xd5, 0xfe, 0xc9, 0x76, - 0x5c, 0x7a, 0xe5, 0xe6, 0x74, 0x61, 0xdf, 0x12, 0xcf, 0xd9, 0xae, 0x2a, 0x13, 0xcf, 0x73, 0x9a, - 0xad, 0x85, 0x8c, 0x0e, 0xa1, 0x7d, 0xa0, 0x0b, 0x66, 0x55, 0xcc, 0x2b, 0x75, 0xe6, 0x34, 0x2f, - 0xbd, 0xde, 0xbc, 0x21, 0x15, 0x54, 0x1f, 0x43, 0x67, 0xa6, 0x90, 0x37, 0x36, 0x92, 0x2b, 0xe3, - 0xb9, 0x15, 0xfe, 0x42, 0xb5, 0x8e, 0x60, 0x65, 0xba, 0x8e, 0x37, 0x9e, 0x57, 0x89, 0x72, 0x7e, - 0x7d, 0xbf, 0x90, 0xd5, 0xbb, 0x50, 0xd5, 0xb5, 0x99, 0xa1, 0xae, 0xe6, 0xa7, 0x6a, 0xb5, 0x85, - 0x53, 0xef, 0x43, 0x3d, 0x53, 0x0a, 0xe9, 0x6c, 0x37, 0x5b, 0x1d, 0xf5, 0xd4, 0x4d, 0x7a, 0x42, - 0x79, 0x1f, 0x2a, 0xaa, 0xfc, 0x31, 0xd6, 0x92, 0x4d, 0x9e, 0xa9, 0x86, 0xae, 0xda, 0x61, 0x07, - 0x88, 0x65, 0x8a, 0x1a, 0x2d, 0x74, 0xb6, 0xce, 0xd1, 0x29, 0x36, 0x37, 0xa2, 0xd6, 0x62, 0x07, - 0x1a, 0xd9, 0xb2, 0x46, 0x2f, 0xe9, 0x9c, 0x52, 0x67, 0x91, 0x26, 0xbb, 0x97, 0x5f, 0x7f, 0xb3, - 0xf1, 0xcc, 0xdf, 0xbf, 0xd9, 0x78, 0xe6, 0x57, 0x4f, 0x36, 0x0a, 0x5f, 0x3f, 0xd9, 0x28, 0xfc, - 0xed, 0xc9, 0x46, 0xe1, 0x5f, 0x4f, 0x36, 0x0a, 0x3f, 0xf9, 0xf9, 0xff, 0xf8, 0x1f, 0x21, 0x1a, - 0x07, 0x0c, 0xfb, 0x68, 0xfb, 0x02, 0x53, 0x96, 0x19, 0x0a, 0xcf, 0x87, 0xf2, 0x8f, 0x42, 0x99, - 0xff, 0x0f, 0x71, 0x2d, 0x07, 0xcb, 0x02, 0x7e, 0xeb, 0x3f, 0x01, 0x00, 0x00, 0xff, 0xff, 0x6c, - 0xf4, 0x1d, 0x49, 0x8c, 0x24, 0x00, 0x00, + // 3207 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x3a, 0x4b, 0x73, 0x1b, 0xc7, + 0xd1, 0xc6, 0x83, 0x04, 0xd0, 0x78, 0x11, 0x4b, 0x8a, 0x82, 0x60, 0x9b, 0x96, 0x57, 0xb6, 0x4c, + 0xd9, 0x9f, 0x48, 0x5b, 0x76, 0x59, 0x7e, 0x94, 0x3f, 0x7d, 0x24, 0x45, 0x93, 0xb4, 0x4d, 0x0b, + 0xdf, 0x42, 0x8c, 0x53, 0x49, 0x25, 0x5b, 0xcb, 0xdd, 0x21, 0x38, 0x26, 0x76, 0x67, 0x3d, 0x3b, + 0x0b, 0x91, 0x4e, 0x55, 0x2a, 0xa7, 0xe4, 0x96, 0x63, 0x6e, 0xf9, 0x03, 0xa9, 0xfc, 0x83, 0x5c, + 0x73, 0x50, 0xe5, 0x94, 0x63, 0x2e, 0x49, 0xc5, 0xfa, 0x09, 0xf9, 0x05, 0xa9, 0x79, 0xed, 0x03, + 0x0f, 0xda, 0x61, 0xa9, 0x2a, 0x17, 0xd4, 0x76, 0x4f, 0x4f, 0xbf, 0xa6, 0xa7, 0xa7, 0x7b, 0x06, + 0xd0, 0x1f, 0x62, 0x76, 0x1a, 0x1f, 0x6f, 0xb8, 0xc4, 0xdf, 0x3c, 0x73, 0x98, 0x73, 0xd7, 0x25, + 0x01, 0x73, 0x70, 0x80, 0x68, 0x34, 0x05, 0x47, 0xd4, 0xdd, 0x1c, 0xe1, 0xe3, 0x68, 0x33, 0xa4, + 0x84, 0x11, 0x97, 0x8c, 0xd4, 0x57, 0xb4, 0xe9, 0x0c, 0x51, 0xc0, 0x36, 0x04, 0x60, 0x94, 0x87, + 0x34, 0x74, 0x7b, 0x35, 0xe2, 0x62, 0x89, 0xe8, 0xd5, 0xdc, 0x48, 0x7f, 0xd6, 0xd9, 0x45, 0x88, + 0x22, 0x05, 0xbc, 0x38, 0x24, 0x64, 0x38, 0x42, 0x92, 0xc7, 0x71, 0x7c, 0xb2, 0x89, 0xfc, 0x90, + 0x5d, 0xc8, 0x41, 0xf3, 0xf7, 0x45, 0x58, 0xdd, 0xa1, 0xc8, 0x61, 0x68, 0x47, 0x2b, 0x60, 0xa1, + 0x6f, 0x62, 0x14, 0x31, 0xe3, 0x55, 0x68, 0x24, 0x4a, 0xd9, 0xd8, 0xeb, 0x16, 0x6e, 0x16, 0xd6, + 0x6b, 0x56, 0x3d, 0xc1, 0x1d, 0x78, 0xc6, 0x75, 0xa8, 0xa0, 0x73, 0xe4, 0xf2, 0xd1, 0xa2, 0x18, + 0x5d, 0xe4, 0xe0, 0x81, 0x67, 0xbc, 0x03, 0xf5, 0x88, 0x51, 0x1c, 0x0c, 0xed, 0x38, 0x42, 0xb4, + 0x5b, 0xba, 0x59, 0x58, 0xaf, 0xdf, 0x5b, 0xda, 0xe0, 0x2a, 0x6f, 0x0c, 0xc4, 0xc0, 0x51, 0x84, + 0xa8, 0x05, 0x51, 0xf2, 0x6d, 0xdc, 0x86, 0x8a, 0x87, 0xc6, 0xd8, 0x45, 0x51, 0xb7, 0x7c, 0xb3, + 0xb4, 0x5e, 0xbf, 0xd7, 0x90, 0xe4, 0x0f, 0x05, 0xd2, 0xd2, 0x83, 0xc6, 0x1d, 0xa8, 0x46, 0x8c, + 0x50, 0x67, 0x88, 0xa2, 0xee, 0x82, 0x20, 0x6c, 0x6a, 0xbe, 0x02, 0x6b, 0x25, 0xc3, 0xc6, 0x4b, + 0x50, 0x7a, 0xb4, 0x73, 0xd0, 0x5d, 0x14, 0xd2, 0x41, 0x51, 0x85, 0xc8, 0xb5, 0x38, 0xda, 0xb8, + 0x05, 0xcd, 0xc8, 0x09, 0xbc, 0x63, 0x72, 0x6e, 0x87, 0xd8, 0x0b, 0xa2, 0x6e, 0xe5, 0x66, 0x61, + 0xbd, 0x6a, 0x35, 0x14, 0xb2, 0xcf, 0x71, 0xe6, 0x47, 0x70, 0x6d, 0xc0, 0x1c, 0xca, 0xae, 0xe0, + 0x1d, 0xf3, 0x08, 0x56, 0x2d, 0xe4, 0x93, 0xf1, 0x95, 0x5c, 0xdb, 0x85, 0x0a, 0xc3, 0x3e, 0x22, + 0x31, 0x13, 0xae, 0x6d, 0x5a, 0x1a, 0x34, 0xff, 0x58, 0x00, 0x63, 0xf7, 0x1c, 0xb9, 0x7d, 0x4a, + 0x5c, 0x14, 0x45, 0xff, 0xa5, 0xe5, 0x7a, 0x03, 0x2a, 0xa1, 0x54, 0xa0, 0x5b, 0x16, 0xe4, 0x6a, + 0x15, 0xb4, 0x56, 0x7a, 0xd4, 0xfc, 0x1a, 0x56, 0x06, 0x78, 0x18, 0x38, 0xa3, 0xe7, 0xa8, 0xef, + 0x2a, 0x2c, 0x46, 0x82, 0xa7, 0x50, 0xb5, 0x69, 0x29, 0xc8, 0xec, 0x83, 0xf1, 0x95, 0x83, 0xd9, + 0xf3, 0x93, 0x64, 0xde, 0x85, 0xe5, 0x1c, 0xc7, 0x28, 0x24, 0x41, 0x84, 0x84, 0x02, 0xcc, 0x61, + 0x71, 0x24, 0x98, 0x2d, 0x58, 0x0a, 0x32, 0x09, 0xac, 0x1e, 0x85, 0xde, 0x15, 0x77, 0xd3, 0x3d, + 0xa8, 0x51, 0x14, 0x91, 0x98, 0xf2, 0x3d, 0x50, 0x14, 0x4e, 0x5d, 0x91, 0x4e, 0xfd, 0x02, 0x07, + 0xf1, 0xb9, 0xa5, 0xc7, 0xac, 0x94, 0x4c, 0xc5, 0x27, 0x8b, 0xae, 0x12, 0x9f, 0x1f, 0xc1, 0xb5, + 0xbe, 0x13, 0x47, 0x57, 0xd1, 0xd5, 0xfc, 0x98, 0xc7, 0x76, 0x14, 0xfb, 0x57, 0x9a, 0xfc, 0x87, + 0x02, 0x54, 0x77, 0xc2, 0xf8, 0x28, 0x72, 0x86, 0xc8, 0x78, 0x05, 0xea, 0x8c, 0x30, 0x67, 0x64, + 0xc7, 0x1c, 0x14, 0xe4, 0x65, 0x0b, 0x04, 0x4a, 0x12, 0xbc, 0x0a, 0x8d, 0x10, 0x51, 0x37, 0x8c, + 0x15, 0x45, 0xf1, 0x66, 0x69, 0xbd, 0x6c, 0xd5, 0x25, 0x4e, 0x92, 0x6c, 0xc0, 0xb2, 0x18, 0xb3, + 0x71, 0x60, 0x9f, 0x21, 0x1a, 0xa0, 0x91, 0x4f, 0x3c, 0x24, 0x82, 0xa3, 0x6c, 0x75, 0xc4, 0xd0, + 0x41, 0xf0, 0x79, 0x32, 0x60, 0xbc, 0x09, 0x9d, 0x84, 0x9e, 0x47, 0xbc, 0xa0, 0x2e, 0x0b, 0xea, + 0xb6, 0xa2, 0x3e, 0x52, 0x68, 0xf3, 0x97, 0xd0, 0x7a, 0x7c, 0x4a, 0x09, 0x63, 0x23, 0x1c, 0x0c, + 0x1f, 0x3a, 0xcc, 0xe1, 0x5b, 0x33, 0x44, 0x14, 0x13, 0x2f, 0x52, 0xda, 0x6a, 0xd0, 0x78, 0x0b, + 0x3a, 0x4c, 0xd2, 0x22, 0xcf, 0xd6, 0x34, 0x45, 0x41, 0xb3, 0x94, 0x0c, 0xf4, 0x15, 0xf1, 0xeb, + 0xd0, 0x4a, 0x89, 0xf9, 0xe6, 0x56, 0xfa, 0x36, 0x13, 0xec, 0x63, 0xec, 0x23, 0x73, 0x2c, 0x7c, + 0x25, 0x16, 0xd9, 0x78, 0x0b, 0x6a, 0xa9, 0x1f, 0x0a, 0x22, 0x42, 0x5a, 0x32, 0x42, 0xb4, 0x3b, + 0xad, 0x6a, 0xe2, 0x94, 0x4f, 0xa0, 0xcd, 0x12, 0xc5, 0x6d, 0xcf, 0x61, 0x4e, 0x3e, 0xa8, 0xf2, + 0x56, 0x59, 0x2d, 0x96, 0x83, 0xcd, 0x8f, 0xa1, 0xd6, 0xc7, 0x5e, 0x24, 0x05, 0x77, 0xa1, 0xe2, + 0xc6, 0x94, 0xa2, 0x80, 0x69, 0x93, 0x15, 0x68, 0xac, 0xc0, 0xc2, 0x08, 0xfb, 0x98, 0x29, 0x33, + 0x25, 0x60, 0x12, 0x80, 0x43, 0xe4, 0x13, 0x7a, 0x21, 0x1c, 0xb6, 0x02, 0x0b, 0xd9, 0xc5, 0x95, + 0x80, 0xf1, 0x22, 0xd4, 0x7c, 0xe7, 0x3c, 0x59, 0x54, 0x3e, 0x52, 0xf5, 0x9d, 0x73, 0xa9, 0x7c, + 0x17, 0x2a, 0x27, 0x0e, 0x1e, 0xb9, 0x01, 0x53, 0x5e, 0xd1, 0x60, 0x2a, 0xb0, 0x9c, 0x15, 0xf8, + 0xe7, 0x22, 0xd4, 0xa5, 0x44, 0xa9, 0xf0, 0x0a, 0x2c, 0xb8, 0x8e, 0x7b, 0x9a, 0x88, 0x14, 0x80, + 0x71, 0x5b, 0x2b, 0x52, 0xcc, 0x66, 0xb8, 0x54, 0x53, 0xad, 0xda, 0x26, 0x40, 0xf4, 0xc4, 0x09, + 0x95, 0x6e, 0xa5, 0x39, 0xc4, 0x35, 0x4e, 0x23, 0xd5, 0x7d, 0x17, 0x1a, 0x32, 0xee, 0xd4, 0x94, + 0xf2, 0x9c, 0x29, 0x75, 0x49, 0x25, 0x27, 0xdd, 0x82, 0x66, 0x1c, 0x21, 0xfb, 0x14, 0x23, 0xea, + 0x50, 0xf7, 0xf4, 0xa2, 0xbb, 0x20, 0x0f, 0xa0, 0x38, 0x42, 0xfb, 0x1a, 0x67, 0xdc, 0x83, 0x05, + 0x9e, 0x5b, 0xa2, 0xee, 0xa2, 0x38, 0xeb, 0x5e, 0xca, 0xb2, 0x14, 0xa6, 0x6e, 0x88, 0xdf, 0xdd, + 0x80, 0xd1, 0x0b, 0x4b, 0x92, 0xf6, 0x3e, 0x00, 0x48, 0x91, 0xc6, 0x12, 0x94, 0xce, 0xd0, 0x85, + 0xda, 0x87, 0xfc, 0x93, 0x3b, 0x67, 0xec, 0x8c, 0x62, 0xed, 0x75, 0x09, 0x7c, 0x54, 0xfc, 0xa0, + 0x60, 0xba, 0xd0, 0xde, 0x1e, 0x9d, 0x61, 0x92, 0x99, 0xbe, 0x02, 0x0b, 0xbe, 0xf3, 0x35, 0xa1, + 0xda, 0x93, 0x02, 0x10, 0x58, 0x1c, 0x10, 0xaa, 0x59, 0x08, 0xc0, 0x68, 0x41, 0x91, 0x84, 0xc2, + 0x5f, 0x35, 0xab, 0x48, 0xc2, 0x54, 0x50, 0x39, 0x23, 0xc8, 0xfc, 0x47, 0x19, 0x20, 0x95, 0x62, + 0x58, 0xd0, 0xc3, 0xc4, 0x8e, 0x10, 0xe5, 0xe7, 0xbb, 0x7d, 0x7c, 0xc1, 0x50, 0x64, 0x53, 0xe4, + 0xc6, 0x34, 0xc2, 0x63, 0xbe, 0x7e, 0xdc, 0xec, 0x6b, 0xd2, 0xec, 0x09, 0xdd, 0xac, 0xeb, 0x98, + 0x0c, 0xe4, 0xbc, 0x6d, 0x3e, 0xcd, 0xd2, 0xb3, 0x8c, 0x03, 0xb8, 0x96, 0xf2, 0xf4, 0x32, 0xec, + 0x8a, 0x97, 0xb1, 0x5b, 0x4e, 0xd8, 0x79, 0x29, 0xab, 0x5d, 0x58, 0xc6, 0xc4, 0xfe, 0x26, 0x46, + 0x71, 0x8e, 0x51, 0xe9, 0x32, 0x46, 0x1d, 0x4c, 0xfe, 0x5f, 0x4c, 0x48, 0xd9, 0xf4, 0xe1, 0x46, + 0xc6, 0x4a, 0xbe, 0xdd, 0x33, 0xcc, 0xca, 0x97, 0x31, 0x5b, 0x4d, 0xb4, 0xe2, 0xf9, 0x20, 0xe5, + 0xf8, 0x19, 0xac, 0x62, 0x62, 0x3f, 0x71, 0x30, 0x9b, 0x64, 0xb7, 0xf0, 0x3d, 0x46, 0xf2, 0x13, + 0x2d, 0xcf, 0x4b, 0x1a, 0xe9, 0x23, 0x3a, 0xcc, 0x19, 0xb9, 0xf8, 0x3d, 0x46, 0x1e, 0x8a, 0x09, + 0x29, 0x9b, 0x2d, 0xe8, 0x60, 0x32, 0xa9, 0x4d, 0xe5, 0x32, 0x26, 0x6d, 0x4c, 0xf2, 0x9a, 0x6c, + 0x43, 0x27, 0x42, 0x2e, 0x23, 0x34, 0x1b, 0x04, 0xd5, 0xcb, 0x58, 0x2c, 0x29, 0xfa, 0x84, 0x87, + 0xf9, 0x53, 0x68, 0xec, 0xc7, 0x43, 0xc4, 0x46, 0xc7, 0x49, 0x32, 0x78, 0x6e, 0xf9, 0xc7, 0xfc, + 0x57, 0x11, 0xea, 0x3b, 0x43, 0x4a, 0xe2, 0x30, 0x97, 0x93, 0xe5, 0x26, 0x9d, 0xcc, 0xc9, 0x82, + 0x44, 0xe4, 0x64, 0x49, 0xfc, 0x1e, 0x34, 0x7c, 0xb1, 0x75, 0x15, 0xbd, 0xcc, 0x43, 0x9d, 0xa9, + 0x4d, 0x6d, 0xd5, 0xfd, 0x4c, 0x32, 0xdb, 0x00, 0x08, 0xb1, 0x17, 0xa9, 0x39, 0x32, 0x1d, 0xb5, + 0x55, 0xb9, 0xa5, 0x53, 0xb4, 0x55, 0x0b, 0x93, 0x6c, 0xfd, 0x0e, 0xd4, 0x8f, 0xb9, 0x93, 0xd4, + 0x84, 0x5c, 0x32, 0x4a, 0xbd, 0x67, 0xc1, 0x71, 0xba, 0x09, 0xf7, 0xa1, 0x79, 0x2a, 0x5d, 0xa6, + 0x26, 0xc9, 0x18, 0xba, 0xa5, 0x2c, 0x49, 0xed, 0xdd, 0xc8, 0x7a, 0x56, 0x2e, 0x40, 0xe3, 0x34, + 0x83, 0xea, 0x0d, 0xa0, 0x33, 0x45, 0x32, 0x23, 0x07, 0xad, 0x67, 0x73, 0x50, 0xfd, 0x9e, 0x21, + 0x05, 0x65, 0x67, 0x66, 0xf3, 0xd2, 0x6f, 0x8b, 0xd0, 0xf8, 0x12, 0xb1, 0x27, 0x84, 0x9e, 0x49, + 0x7d, 0x0d, 0x28, 0x07, 0x8e, 0x8f, 0x14, 0x47, 0xf1, 0x6d, 0xdc, 0x80, 0x2a, 0x3d, 0x97, 0x09, + 0x44, 0xad, 0x67, 0x85, 0x9e, 0x8b, 0xc4, 0x60, 0xbc, 0x0c, 0x40, 0xcf, 0xed, 0xd0, 0x71, 0xcf, + 0x90, 0xf2, 0x60, 0xd9, 0xaa, 0xd1, 0xf3, 0xbe, 0x44, 0xf0, 0x50, 0xa0, 0xe7, 0x36, 0xa2, 0x94, + 0xd0, 0x48, 0xe5, 0xaa, 0x2a, 0x3d, 0xdf, 0x15, 0xb0, 0x9a, 0xeb, 0x51, 0x12, 0x86, 0xc8, 0x13, + 0x39, 0x5a, 0xcc, 0x7d, 0x28, 0x11, 0x5c, 0x2a, 0xd3, 0x52, 0x17, 0xa5, 0x54, 0x96, 0x4a, 0x65, + 0xa9, 0xd4, 0x8a, 0x9c, 0xc9, 0xb2, 0x52, 0x59, 0x22, 0xb5, 0x2a, 0xa5, 0xb2, 0x8c, 0x54, 0x96, + 0x4a, 0xad, 0xe9, 0xb9, 0x4a, 0xaa, 0xf9, 0x9b, 0x02, 0xac, 0x4e, 0x16, 0x7e, 0xaa, 0x36, 0x7d, + 0x0f, 0x1a, 0xae, 0x58, 0xaf, 0x5c, 0x4c, 0x76, 0xa6, 0x56, 0xd2, 0xaa, 0xbb, 0x99, 0x30, 0xbe, + 0x0f, 0xcd, 0x40, 0x3a, 0x38, 0x09, 0xcd, 0x52, 0xba, 0x2e, 0x59, 0xdf, 0x5b, 0x8d, 0x20, 0x03, + 0x99, 0x1e, 0x18, 0x5f, 0x51, 0xcc, 0xd0, 0x80, 0x51, 0xe4, 0xf8, 0xcf, 0xa3, 0xba, 0x37, 0xa0, + 0x2c, 0xaa, 0x15, 0xbe, 0x4c, 0x0d, 0x4b, 0x7c, 0x9b, 0x6f, 0xc0, 0x72, 0x4e, 0x8a, 0xb2, 0x75, + 0x09, 0x4a, 0x23, 0x14, 0x08, 0xee, 0x4d, 0x8b, 0x7f, 0x9a, 0x0e, 0x74, 0x2c, 0xe4, 0x78, 0xcf, + 0x4f, 0x1b, 0x25, 0xa2, 0x94, 0x8a, 0x58, 0x07, 0x23, 0x2b, 0x42, 0xa9, 0xa2, 0xb5, 0x2e, 0x64, + 0xb4, 0x7e, 0x04, 0x9d, 0x9d, 0x11, 0x89, 0xd0, 0x80, 0x79, 0x38, 0x78, 0x1e, 0xed, 0xc8, 0x2f, + 0x60, 0xf9, 0x31, 0xbb, 0xf8, 0x8a, 0x33, 0x8b, 0xf0, 0xb7, 0xe8, 0x39, 0xd9, 0x47, 0xc9, 0x13, + 0x6d, 0x1f, 0x25, 0x4f, 0x78, 0x73, 0xe3, 0x92, 0x51, 0xec, 0x07, 0x62, 0x2b, 0x34, 0x2d, 0x05, + 0x99, 0xdb, 0xd0, 0x90, 0x35, 0xf4, 0x21, 0xf1, 0xe2, 0x11, 0x9a, 0xb9, 0x07, 0xd7, 0x00, 0x42, + 0x87, 0x3a, 0x3e, 0x62, 0x88, 0xca, 0x18, 0xaa, 0x59, 0x19, 0x8c, 0xf9, 0xbb, 0x22, 0xac, 0xc8, + 0xfb, 0x86, 0x81, 0x6c, 0xb3, 0xb5, 0x09, 0x3d, 0xa8, 0x9e, 0x92, 0x88, 0x65, 0x18, 0x26, 0x30, + 0x57, 0x91, 0xf7, 0xe7, 0x92, 0x1b, 0xff, 0xcc, 0x5d, 0x02, 0x94, 0x2e, 0xbf, 0x04, 0x98, 0x6a, + 0xf3, 0xcb, 0xd3, 0x6d, 0x3e, 0xdf, 0x6d, 0x9a, 0x08, 0xcb, 0x3d, 0x5e, 0xb3, 0x6a, 0x0a, 0x73, + 0xe0, 0x19, 0xb7, 0xa1, 0x3d, 0xe4, 0x5a, 0xda, 0xa7, 0x84, 0x9c, 0xd9, 0xa1, 0xc3, 0x4e, 0xc5, + 0x56, 0xaf, 0x59, 0x4d, 0x81, 0xde, 0x27, 0xe4, 0xac, 0xef, 0xb0, 0x53, 0xe3, 0x43, 0x68, 0xa9, + 0x32, 0xd0, 0x17, 0x2e, 0x8a, 0xd4, 0xe1, 0xa7, 0x76, 0x51, 0xd6, 0x7b, 0x56, 0xf3, 0x2c, 0x03, + 0x45, 0xe6, 0x75, 0xb8, 0xf6, 0x10, 0x45, 0x8c, 0x92, 0x8b, 0xbc, 0x63, 0xcc, 0xff, 0x05, 0x38, + 0x08, 0x18, 0xa2, 0x27, 0x8e, 0x8b, 0x22, 0xe3, 0xed, 0x2c, 0xa4, 0x8a, 0xa3, 0xa5, 0x0d, 0x79, + 0xdd, 0x93, 0x0c, 0x58, 0x19, 0x1a, 0x73, 0x03, 0x16, 0x2d, 0x12, 0xf3, 0x74, 0xf4, 0x9a, 0xfe, + 0x52, 0xf3, 0x1a, 0x6a, 0x9e, 0x40, 0x5a, 0x6a, 0xcc, 0xdc, 0xd7, 0x2d, 0x6c, 0xca, 0x4e, 0x2d, + 0xd1, 0x06, 0xd4, 0xb0, 0xc6, 0xa9, 0xac, 0x32, 0x2d, 0x3a, 0x25, 0x31, 0x3f, 0x86, 0x65, 0xc9, + 0x49, 0x72, 0xd6, 0x6c, 0x5e, 0x83, 0x45, 0xaa, 0xd5, 0x28, 0xa4, 0xf7, 0x3c, 0x8a, 0x48, 0x8d, + 0x71, 0x7f, 0x7c, 0x81, 0x23, 0x96, 0x1a, 0xa2, 0xfd, 0xb1, 0x0c, 0x1d, 0x3e, 0x90, 0xe3, 0x69, + 0x7e, 0x0a, 0x8d, 0x2d, 0xab, 0xff, 0x25, 0xc2, 0xc3, 0xd3, 0x63, 0x9e, 0x3d, 0xdf, 0xcf, 0xc3, + 0xca, 0x60, 0x43, 0x69, 0x9b, 0x19, 0xb2, 0x72, 0x74, 0xe6, 0x67, 0xb0, 0xba, 0xe5, 0x79, 0x59, + 0x94, 0xd6, 0xfa, 0x6d, 0xa8, 0x05, 0x19, 0x76, 0x99, 0x33, 0x2b, 0x47, 0x9d, 0x12, 0x99, 0x77, + 0xc1, 0xd8, 0x43, 0xec, 0xa0, 0xff, 0xd8, 0x39, 0x1e, 0xa5, 0xd6, 0x5f, 0x87, 0x0a, 0x8e, 0x6c, + 0x1c, 0x8e, 0xdf, 0x17, 0x5c, 0xaa, 0xd6, 0x22, 0x8e, 0x0e, 0xc2, 0xf1, 0xfb, 0xe6, 0x1d, 0x58, + 0xce, 0x91, 0x5f, 0x92, 0x56, 0xb6, 0xc0, 0x18, 0xfc, 0x70, 0xce, 0x09, 0x8b, 0x62, 0x86, 0xc5, + 0x1d, 0x58, 0x1e, 0xfc, 0x40, 0x69, 0x3f, 0x83, 0xe5, 0x47, 0xc1, 0x08, 0x07, 0x68, 0xa7, 0x7f, + 0x74, 0x88, 0x92, 0x9c, 0x6a, 0x40, 0x99, 0xd7, 0x9e, 0x4a, 0x96, 0xf8, 0xe6, 0x2a, 0x04, 0xc7, + 0xb6, 0x1b, 0xc6, 0x91, 0xba, 0xb4, 0x5a, 0x0c, 0x8e, 0x77, 0xc2, 0x38, 0xe2, 0x87, 0x24, 0x2f, + 0x92, 0x48, 0x30, 0xba, 0x10, 0x99, 0xa6, 0x6a, 0x55, 0xdc, 0x30, 0x7e, 0x14, 0x8c, 0x2e, 0xcc, + 0xff, 0x11, 0x37, 0x09, 0x08, 0x79, 0x96, 0x13, 0x78, 0xc4, 0x7f, 0x88, 0xc6, 0x19, 0x09, 0x53, + 0x7a, 0x3f, 0x2d, 0x40, 0x63, 0x6b, 0x88, 0x02, 0xf6, 0x10, 0x31, 0x07, 0x8f, 0x44, 0x67, 0x3a, + 0x46, 0x34, 0xc2, 0x24, 0x50, 0x69, 0x43, 0x83, 0xc6, 0x2b, 0x50, 0xc7, 0x01, 0x66, 0xb6, 0xe7, + 0x20, 0x9f, 0x04, 0x82, 0x4b, 0xd5, 0x02, 0x8e, 0x7a, 0x28, 0x30, 0xc6, 0x1b, 0xd0, 0x96, 0x97, + 0x8a, 0xf6, 0xa9, 0x13, 0x78, 0x23, 0x9e, 0xb0, 0x4a, 0x22, 0xc5, 0xb4, 0x24, 0x7a, 0x5f, 0x61, + 0x8d, 0x3b, 0xb0, 0xa4, 0xd2, 0x49, 0x4a, 0x59, 0x16, 0x94, 0x6d, 0x85, 0xcf, 0x91, 0xc6, 0x61, + 0x48, 0x28, 0x8b, 0xec, 0x08, 0xb9, 0x2e, 0xf1, 0x43, 0xd5, 0xd6, 0xb5, 0x35, 0x7e, 0x20, 0xd1, + 0xe6, 0x10, 0x96, 0xf7, 0xb8, 0x9d, 0xca, 0x92, 0x74, 0x7b, 0xb4, 0x7c, 0xe4, 0xdb, 0xc7, 0x23, + 0xe2, 0x9e, 0xd9, 0x3c, 0xc9, 0x2b, 0x0f, 0xf3, 0xc2, 0x71, 0x9b, 0x23, 0x07, 0xf8, 0x5b, 0x71, + 0x83, 0xc1, 0xa9, 0x4e, 0x09, 0x0b, 0x47, 0xf1, 0xd0, 0x0e, 0x29, 0x39, 0x46, 0xca, 0xc4, 0xb6, + 0x8f, 0xfc, 0x7d, 0x89, 0xef, 0x73, 0xb4, 0xf9, 0xa7, 0x02, 0xac, 0xe4, 0x25, 0xa9, 0xd5, 0xde, + 0x84, 0x95, 0xbc, 0x28, 0x55, 0xc6, 0xc8, 0x32, 0xb9, 0x93, 0x15, 0x28, 0x0b, 0x9a, 0xfb, 0xd0, + 0x14, 0x57, 0xd0, 0xb6, 0x27, 0x39, 0xe5, 0x8b, 0xb7, 0xec, 0xba, 0x58, 0x0d, 0x27, 0xbb, 0x4a, + 0x1f, 0xc2, 0x0d, 0x65, 0xbe, 0x3d, 0xad, 0xb6, 0x0c, 0x88, 0x55, 0x45, 0x70, 0x38, 0xa1, 0xfd, + 0x17, 0xd0, 0x4d, 0x51, 0xdb, 0x17, 0x02, 0x99, 0x6e, 0xca, 0xe5, 0x09, 0x63, 0xb7, 0x3c, 0x8f, + 0x8a, 0xdd, 0x5e, 0xb6, 0x66, 0x0d, 0x99, 0x0f, 0xe0, 0xfa, 0x00, 0x31, 0xe9, 0x0d, 0x87, 0xa9, + 0x8e, 0x4a, 0x32, 0x5b, 0x82, 0xd2, 0x00, 0xb9, 0xc2, 0xf8, 0x92, 0xc5, 0x3f, 0x79, 0x00, 0x1e, + 0x45, 0xc8, 0x15, 0x56, 0x96, 0x2c, 0xf1, 0x6d, 0x86, 0x50, 0xf9, 0x74, 0xb0, 0xc7, 0xeb, 0x26, + 0x1e, 0xd4, 0xb2, 0xce, 0x52, 0x67, 0x6a, 0xd3, 0xaa, 0x08, 0xf8, 0xc0, 0x33, 0x3e, 0x83, 0x65, + 0x39, 0xe4, 0x9e, 0x3a, 0xc1, 0x10, 0xd9, 0x21, 0x19, 0x61, 0x57, 0x86, 0x7e, 0xeb, 0x5e, 0x4f, + 0xa5, 0x21, 0xc5, 0x67, 0x47, 0x90, 0xf4, 0x05, 0x85, 0xd5, 0x19, 0x4e, 0xa2, 0xcc, 0xbf, 0x17, + 0xa0, 0xa2, 0x8e, 0x35, 0x7e, 0x34, 0x7b, 0x14, 0x8f, 0x11, 0x55, 0xc1, 0xae, 0x20, 0xe3, 0x75, + 0x68, 0xc9, 0x2f, 0x9b, 0x84, 0x0c, 0x93, 0xe4, 0xb0, 0x6c, 0x4a, 0xec, 0x23, 0x89, 0x14, 0xd7, + 0x96, 0xe2, 0xe2, 0x50, 0xf5, 0xe8, 0x0a, 0xe2, 0xf8, 0x93, 0x88, 0x2b, 0x25, 0x0e, 0xc7, 0x9a, + 0xa5, 0x20, 0xbe, 0xb9, 0x34, 0xbf, 0x05, 0xc1, 0x4f, 0x83, 0x7c, 0x73, 0xf9, 0x24, 0x0e, 0x98, + 0x1d, 0x12, 0x1c, 0x30, 0x75, 0x1a, 0x82, 0x40, 0xf5, 0x39, 0xc6, 0x58, 0x87, 0xea, 0x49, 0x64, + 0x0b, 0x6b, 0x44, 0xe5, 0x9b, 0x9c, 0xd0, 0xca, 0x6a, 0xab, 0x72, 0x12, 0x89, 0x0f, 0xf3, 0xd7, + 0x05, 0x58, 0x94, 0x97, 0xfc, 0x46, 0x0b, 0x8a, 0x49, 0xf5, 0x52, 0xc4, 0xa2, 0x12, 0x14, 0x5a, + 0xc9, 0x8a, 0x45, 0x7c, 0xf3, 0x1c, 0x33, 0xf6, 0xe5, 0x19, 0xac, 0x8c, 0x18, 0xfb, 0xe2, 0xf0, + 0x7d, 0x1d, 0x5a, 0x69, 0x11, 0x24, 0xc6, 0xa5, 0x31, 0xcd, 0x04, 0x2b, 0xc8, 0xe6, 0xda, 0x64, + 0xfe, 0x18, 0x20, 0xbd, 0xec, 0xe6, 0xe1, 0x10, 0x27, 0xca, 0xf0, 0x4f, 0x8e, 0x19, 0x26, 0xe5, + 0x13, 0xff, 0x34, 0x6e, 0x43, 0xcb, 0xf1, 0x3c, 0xcc, 0xa7, 0x3b, 0xa3, 0x3d, 0xec, 0x25, 0x09, + 0x24, 0x8f, 0x35, 0xff, 0x52, 0x80, 0xf6, 0x0e, 0x09, 0x2f, 0x3e, 0xc5, 0x23, 0x94, 0xc9, 0x6e, + 0x42, 0x49, 0x55, 0x3d, 0xf1, 0x6f, 0xde, 0x11, 0x9c, 0xe0, 0x11, 0x92, 0xdb, 0x5e, 0x46, 0x5d, + 0x95, 0x23, 0xc4, 0x96, 0xd7, 0x83, 0xc9, 0xd5, 0x66, 0x53, 0x0e, 0x1e, 0x12, 0x4f, 0xf4, 0x3e, + 0x1e, 0xa6, 0x76, 0x72, 0x91, 0xd9, 0xb4, 0x2a, 0x1e, 0xa6, 0x62, 0x48, 0x19, 0xb2, 0x20, 0x2e, + 0xaa, 0xb3, 0x86, 0x2c, 0x4a, 0x0c, 0x37, 0x64, 0x15, 0x16, 0xc9, 0xc9, 0x49, 0x84, 0x98, 0x58, + 0xab, 0x92, 0xa5, 0xa0, 0x24, 0x05, 0x57, 0x33, 0x29, 0x78, 0x45, 0x9c, 0x6b, 0x8f, 0x1e, 0x1d, + 0xee, 0x8e, 0x51, 0xc0, 0xf4, 0x09, 0x7c, 0x17, 0xaa, 0x1a, 0xf5, 0x43, 0xae, 0x80, 0xdf, 0x84, + 0xd6, 0x96, 0xe7, 0x0d, 0x9e, 0x38, 0xa1, 0xf6, 0x47, 0x17, 0x2a, 0xfd, 0x9d, 0x83, 0xbe, 0x74, + 0x49, 0x89, 0x1b, 0xa0, 0x40, 0x7e, 0xe2, 0xef, 0x21, 0x76, 0x88, 0x18, 0xc5, 0x6e, 0x72, 0xe2, + 0xdf, 0x82, 0x8a, 0xc2, 0xf0, 0x99, 0xbe, 0xfc, 0xd4, 0x47, 0x80, 0x02, 0xcd, 0xff, 0x03, 0xe3, + 0x47, 0xbc, 0x76, 0x45, 0xb2, 0x71, 0x51, 0x92, 0xde, 0x84, 0xce, 0x58, 0x60, 0x6d, 0x59, 0xd4, + 0x65, 0x96, 0xa1, 0x2d, 0x07, 0x44, 0x7e, 0x10, 0xb2, 0x8f, 0x60, 0x59, 0x96, 0xda, 0x92, 0xcf, + 0x15, 0x58, 0x70, 0x1f, 0x26, 0xeb, 0x59, 0xb6, 0xc4, 0xf7, 0xbd, 0xa7, 0x86, 0x3a, 0xc6, 0xd4, + 0xcd, 0x8e, 0xb1, 0x07, 0xed, 0x89, 0x67, 0x38, 0x43, 0x5d, 0xf5, 0xcd, 0x7e, 0x9d, 0xeb, 0xad, + 0x6e, 0xc8, 0x67, 0xbd, 0x0d, 0xfd, 0xac, 0xb7, 0xb1, 0xeb, 0x87, 0xec, 0xc2, 0xd8, 0x85, 0x56, + 0xfe, 0xc1, 0xca, 0x78, 0x51, 0x57, 0xc6, 0x33, 0x9e, 0xb1, 0xe6, 0xb2, 0xd9, 0x83, 0xf6, 0xc4, + 0xdb, 0x95, 0xd6, 0x67, 0xf6, 0x93, 0xd6, 0x5c, 0x46, 0x0f, 0xa0, 0x9e, 0x79, 0xac, 0x32, 0xba, + 0x92, 0xc9, 0xf4, 0xfb, 0xd5, 0x5c, 0x06, 0x3b, 0xd0, 0xcc, 0xbd, 0x1f, 0x19, 0x3d, 0x65, 0xcf, + 0x8c, 0x47, 0xa5, 0xb9, 0x4c, 0xb6, 0xa1, 0x9e, 0x79, 0xc6, 0xd1, 0x5a, 0x4c, 0xbf, 0x15, 0xf5, + 0x6e, 0xcc, 0x18, 0x51, 0xa7, 0xe5, 0x1e, 0xb4, 0x27, 0xde, 0x76, 0xb4, 0x4b, 0x66, 0x3f, 0xf9, + 0xcc, 0x55, 0xe6, 0x73, 0xb1, 0x44, 0x99, 0xd6, 0x3d, 0xb3, 0x44, 0xd3, 0x2f, 0x39, 0xbd, 0x97, + 0x66, 0x0f, 0x2a, 0xad, 0x76, 0xa1, 0x95, 0x7f, 0xc4, 0xd1, 0xcc, 0x66, 0x3e, 0xed, 0x5c, 0xbe, + 0xde, 0xb9, 0xf7, 0x9c, 0x74, 0xbd, 0x67, 0x3d, 0xf3, 0xcc, 0x65, 0xb4, 0x05, 0xa0, 0x1a, 0x75, + 0x0f, 0x07, 0x89, 0xa3, 0xa7, 0x2e, 0x08, 0x12, 0x47, 0xcf, 0x68, 0xea, 0x1f, 0x00, 0xc8, 0xfe, + 0xda, 0x23, 0x31, 0x33, 0xae, 0x6b, 0x35, 0x26, 0x9a, 0xfa, 0x5e, 0x77, 0x7a, 0x60, 0x8a, 0x01, + 0xa2, 0xf4, 0x2a, 0x0c, 0x3e, 0x01, 0x48, 0xfb, 0x76, 0xcd, 0x60, 0xaa, 0x93, 0xbf, 0xc4, 0x07, + 0x8d, 0x6c, 0x97, 0x6e, 0x28, 0x5b, 0x67, 0x74, 0xee, 0x97, 0xb0, 0x68, 0x4f, 0x74, 0x61, 0xf9, + 0x60, 0x9b, 0x6c, 0xce, 0x7a, 0x53, 0x9d, 0x98, 0x71, 0x1f, 0x1a, 0xd9, 0xf6, 0x4b, 0x6b, 0x31, + 0xa3, 0x25, 0xeb, 0xe5, 0x5a, 0x30, 0xe3, 0x01, 0xb4, 0xf2, 0xad, 0x97, 0x0e, 0xa9, 0x99, 0x0d, + 0x59, 0x4f, 0x5d, 0x2c, 0x66, 0xc8, 0xdf, 0x05, 0x48, 0x5b, 0x34, 0xed, 0xbe, 0xa9, 0xa6, 0x6d, + 0x42, 0xea, 0x1e, 0xb4, 0x27, 0x5a, 0x2f, 0x6d, 0xf1, 0xec, 0x8e, 0xec, 0xb2, 0xbd, 0x9e, 0x69, + 0xa4, 0x74, 0x08, 0x4e, 0xb7, 0x62, 0x3a, 0x04, 0x67, 0x75, 0x5d, 0xdb, 0x50, 0x1f, 0x4c, 0xf3, + 0x18, 0xcc, 0xe5, 0x31, 0xab, 0x97, 0x7a, 0x0f, 0x20, 0x3d, 0xb6, 0xb4, 0x17, 0xa6, 0x0e, 0xb2, + 0x5e, 0x53, 0x5f, 0xfe, 0x4a, 0xba, 0x1d, 0x68, 0xe6, 0xee, 0x47, 0x74, 0xba, 0x9b, 0x75, 0x69, + 0x72, 0xd9, 0x21, 0x90, 0xbf, 0x4c, 0xd0, 0x2b, 0x38, 0xf3, 0x8a, 0xe1, 0xb2, 0x38, 0xce, 0x76, + 0x7e, 0x3a, 0x82, 0x66, 0x74, 0x83, 0xdf, 0x93, 0x57, 0xb2, 0xdd, 0x5d, 0x26, 0xaf, 0xcc, 0x68, + 0xfa, 0xe6, 0x32, 0xda, 0x87, 0xf6, 0x9e, 0x2e, 0xdc, 0x55, 0x53, 0xa1, 0xd7, 0x6f, 0xba, 0x89, + 0xea, 0xf5, 0x66, 0x0d, 0xa9, 0x75, 0xf9, 0x1c, 0x3a, 0x53, 0x0d, 0x85, 0xb1, 0x96, 0x5c, 0xc1, + 0xcf, 0xec, 0x34, 0xe6, 0xaa, 0x75, 0x00, 0x4b, 0x93, 0xfd, 0x84, 0xf1, 0x72, 0x12, 0x13, 0xb3, + 0xfa, 0x8c, 0xb9, 0xac, 0x3e, 0x84, 0xaa, 0xae, 0x11, 0x0d, 0xf5, 0xd4, 0x31, 0x51, 0x33, 0xce, + 0x9d, 0x7a, 0x5f, 0x84, 0x7c, 0x52, 0x7f, 0xa5, 0x21, 0x3f, 0x51, 0xa5, 0xf5, 0xd4, 0xcb, 0x44, + 0x42, 0x79, 0x1f, 0x2a, 0xaa, 0x0c, 0x33, 0x56, 0x92, 0xcd, 0x96, 0xa9, 0xca, 0x2e, 0x8b, 0xb0, + 0x3d, 0xc4, 0x32, 0xc5, 0x95, 0x16, 0x3a, 0x5d, 0x6f, 0xe9, 0x3d, 0x92, 0x1b, 0x51, 0x6b, 0xb1, + 0x05, 0x8d, 0x6c, 0x79, 0xa5, 0x97, 0x74, 0x46, 0xc9, 0x35, 0x4f, 0x93, 0xed, 0xf3, 0xa7, 0xdf, + 0xad, 0xbd, 0xf0, 0xb7, 0xef, 0xd6, 0x5e, 0xf8, 0xd5, 0xb3, 0xb5, 0xc2, 0xd3, 0x67, 0x6b, 0x85, + 0xbf, 0x3e, 0x5b, 0x2b, 0xfc, 0xf3, 0xd9, 0x5a, 0xe1, 0x27, 0x3f, 0xff, 0x0f, 0xff, 0x73, 0x45, + 0xe3, 0x80, 0x61, 0x1f, 0x6d, 0x8e, 0x31, 0x65, 0x99, 0xa1, 0xf0, 0x6c, 0x28, 0xff, 0x78, 0x95, + 0xf9, 0x3f, 0x16, 0xd7, 0xf2, 0x78, 0x51, 0xc0, 0xef, 0xfe, 0x3b, 0x00, 0x00, 0xff, 0xff, 0x77, + 0xb9, 0x27, 0x67, 0xdc, 0x25, 0x00, 0x00, } func (m *CreateContainerRequest) Marshal() (dAtA []byte, err error) { @@ -4822,6 +4992,155 @@ func (m *AddARPNeighborsRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) return len(dAtA) - i, nil } +func (m *GetIPTablesRequest) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *GetIPTablesRequest) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *GetIPTablesRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if m.IsIpv6 { + i-- + if m.IsIpv6 { + dAtA[i] = 1 + } else { + dAtA[i] = 0 + } + i-- + dAtA[i] = 0x8 + } + return len(dAtA) - i, nil +} + +func (m *GetIPTablesResponse) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *GetIPTablesResponse) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *GetIPTablesResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if len(m.Data) > 0 { + i -= len(m.Data) + copy(dAtA[i:], m.Data) + i = encodeVarintAgent(dAtA, i, uint64(len(m.Data))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + +func (m *SetIPTablesRequest) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *SetIPTablesRequest) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *SetIPTablesRequest) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if len(m.Data) > 0 { + i -= len(m.Data) + copy(dAtA[i:], m.Data) + i = encodeVarintAgent(dAtA, i, uint64(len(m.Data))) + i-- + dAtA[i] = 0x12 + } + if m.IsIpv6 { + i-- + if m.IsIpv6 { + dAtA[i] = 1 + } else { + dAtA[i] = 0 + } + i-- + dAtA[i] = 0x8 + } + return len(dAtA) - i, nil +} + +func (m *SetIPTablesResponse) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalToSizedBuffer(dAtA[:size]) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *SetIPTablesResponse) MarshalTo(dAtA []byte) (int, error) { + size := m.Size() + return m.MarshalToSizedBuffer(dAtA[:size]) +} + +func (m *SetIPTablesResponse) MarshalToSizedBuffer(dAtA []byte) (int, error) { + i := len(dAtA) + _ = i + var l int + _ = l + if m.XXX_unrecognized != nil { + i -= len(m.XXX_unrecognized) + copy(dAtA[i:], m.XXX_unrecognized) + } + if len(m.Data) > 0 { + i -= len(m.Data) + copy(dAtA[i:], m.Data) + i = encodeVarintAgent(dAtA, i, uint64(len(m.Data))) + i-- + dAtA[i] = 0xa + } + return len(dAtA) - i, nil +} + func (m *OnlineCPUMemRequest) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) @@ -6640,6 +6959,72 @@ func (m *AddARPNeighborsRequest) Size() (n int) { return n } +func (m *GetIPTablesRequest) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.IsIpv6 { + n += 2 + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *GetIPTablesResponse) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.Data) + if l > 0 { + n += 1 + l + sovAgent(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *SetIPTablesRequest) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + if m.IsIpv6 { + n += 2 + } + l = len(m.Data) + if l > 0 { + n += 1 + l + sovAgent(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + +func (m *SetIPTablesResponse) Size() (n int) { + if m == nil { + return 0 + } + var l int + _ = l + l = len(m.Data) + if l > 0 { + n += 1 + l + sovAgent(uint64(l)) + } + if m.XXX_unrecognized != nil { + n += len(m.XXX_unrecognized) + } + return n +} + func (m *OnlineCPUMemRequest) Size() (n int) { if m == nil { return 0 @@ -7670,6 +8055,51 @@ func (this *AddARPNeighborsRequest) String() string { }, "") return s } +func (this *GetIPTablesRequest) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&GetIPTablesRequest{`, + `IsIpv6:` + fmt.Sprintf("%v", this.IsIpv6) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} +func (this *GetIPTablesResponse) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&GetIPTablesResponse{`, + `Data:` + fmt.Sprintf("%v", this.Data) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} +func (this *SetIPTablesRequest) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&SetIPTablesRequest{`, + `IsIpv6:` + fmt.Sprintf("%v", this.IsIpv6) + `,`, + `Data:` + fmt.Sprintf("%v", this.Data) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} +func (this *SetIPTablesResponse) String() string { + if this == nil { + return "nil" + } + s := strings.Join([]string{`&SetIPTablesResponse{`, + `Data:` + fmt.Sprintf("%v", this.Data) + `,`, + `XXX_unrecognized:` + fmt.Sprintf("%v", this.XXX_unrecognized) + `,`, + `}`, + }, "") + return s +} func (this *OnlineCPUMemRequest) String() string { if this == nil { return "nil" @@ -7938,6 +8368,8 @@ type AgentServiceService interface { ListInterfaces(ctx context.Context, req *ListInterfacesRequest) (*Interfaces, error) ListRoutes(ctx context.Context, req *ListRoutesRequest) (*Routes, error) AddARPNeighbors(ctx context.Context, req *AddARPNeighborsRequest) (*types.Empty, error) + GetIPTables(ctx context.Context, req *GetIPTablesRequest) (*GetIPTablesResponse, error) + SetIPTables(ctx context.Context, req *SetIPTablesRequest) (*SetIPTablesResponse, error) GetMetrics(ctx context.Context, req *GetMetricsRequest) (*Metrics, error) CreateSandbox(ctx context.Context, req *CreateSandboxRequest) (*types.Empty, error) DestroySandbox(ctx context.Context, req *DestroySandboxRequest) (*types.Empty, error) @@ -8095,6 +8527,20 @@ func RegisterAgentServiceService(srv *github_com_containerd_ttrpc.Server, svc Ag } return svc.AddARPNeighbors(ctx, &req) }, + "GetIPTables": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { + var req GetIPTablesRequest + if err := unmarshal(&req); err != nil { + return nil, err + } + return svc.GetIPTables(ctx, &req) + }, + "SetIPTables": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { + var req SetIPTablesRequest + if err := unmarshal(&req); err != nil { + return nil, err + } + return svc.SetIPTables(ctx, &req) + }, "GetMetrics": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { var req GetMetricsRequest if err := unmarshal(&req); err != nil { @@ -8359,6 +8805,22 @@ func (c *agentServiceClient) AddARPNeighbors(ctx context.Context, req *AddARPNei return &resp, nil } +func (c *agentServiceClient) GetIPTables(ctx context.Context, req *GetIPTablesRequest) (*GetIPTablesResponse, error) { + var resp GetIPTablesResponse + if err := c.client.Call(ctx, "grpc.AgentService", "GetIPTables", req, &resp); err != nil { + return nil, err + } + return &resp, nil +} + +func (c *agentServiceClient) SetIPTables(ctx context.Context, req *SetIPTablesRequest) (*SetIPTablesResponse, error) { + var resp SetIPTablesResponse + if err := c.client.Call(ctx, "grpc.AgentService", "SetIPTables", req, &resp); err != nil { + return nil, err + } + return &resp, nil +} + func (c *agentServiceClient) GetMetrics(ctx context.Context, req *GetMetricsRequest) (*Metrics, error) { var resp Metrics if err := c.client.Call(ctx, "grpc.AgentService", "GetMetrics", req, &resp); err != nil { @@ -13746,6 +14208,352 @@ func (m *AddARPNeighborsRequest) Unmarshal(dAtA []byte) error { } return nil } +func (m *GetIPTablesRequest) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: GetIPTablesRequest: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: GetIPTablesRequest: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field IsIpv6", wireType) + } + var v int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + m.IsIpv6 = bool(v != 0) + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *GetIPTablesResponse) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: GetIPTablesResponse: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: GetIPTablesResponse: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAgent + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAgent + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *SetIPTablesRequest) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: SetIPTablesRequest: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: SetIPTablesRequest: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field IsIpv6", wireType) + } + var v int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + m.IsIpv6 = bool(v != 0) + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAgent + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAgent + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *SetIPTablesResponse) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: SetIPTablesResponse: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: SetIPTablesResponse: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Data", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowAgent + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= int(b&0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthAgent + } + postIndex := iNdEx + byteLen + if postIndex < 0 { + return ErrInvalidLengthAgent + } + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Data = append(m.Data[:0], dAtA[iNdEx:postIndex]...) + if m.Data == nil { + m.Data = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipAgent(dAtA[iNdEx:]) + if err != nil { + return err + } + if (skippy < 0) || (iNdEx+skippy) < 0 { + return ErrInvalidLengthAgent + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + m.XXX_unrecognized = append(m.XXX_unrecognized, dAtA[iNdEx:iNdEx+skippy]...) + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} func (m *OnlineCPUMemRequest) Unmarshal(dAtA []byte) error { l := len(dAtA) iNdEx := 0 From 7c4049aabbe1ec7c015bc05200f686351631bbcb Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Tue, 31 May 2022 09:10:38 -0700 Subject: [PATCH 12/35] osbuilder: add iptables package Since we are introducing an agent API for interacting with guest iptables, let's ensure that our example rootfs' have iptables-save/restore installed. Fixes: #4356 Signed-off-by: Eric Ernst --- tools/osbuilder/rootfs-builder/alpine/config.sh | 2 +- tools/osbuilder/rootfs-builder/centos/config.sh | 2 +- tools/osbuilder/rootfs-builder/clearlinux/config.sh | 4 ++-- tools/osbuilder/rootfs-builder/debian/config.sh | 2 +- tools/osbuilder/rootfs-builder/ubuntu/config.sh | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tools/osbuilder/rootfs-builder/alpine/config.sh b/tools/osbuilder/rootfs-builder/alpine/config.sh index 105ecd8fb..1df524333 100644 --- a/tools/osbuilder/rootfs-builder/alpine/config.sh +++ b/tools/osbuilder/rootfs-builder/alpine/config.sh @@ -13,7 +13,7 @@ BASE_PACKAGES="alpine-base" # See a list of mirrors at http://nl.alpinelinux.org/alpine/MIRRORS.txt MIRROR=https://dl-5.alpinelinux.org/alpine -PACKAGES="" +PACKAGES="iptables ip6tables" # Init process must be one of {systemd,kata-agent} INIT_PROCESS=kata-agent diff --git a/tools/osbuilder/rootfs-builder/centos/config.sh b/tools/osbuilder/rootfs-builder/centos/config.sh index d9d51e5d7..7226da047 100644 --- a/tools/osbuilder/rootfs-builder/centos/config.sh +++ b/tools/osbuilder/rootfs-builder/centos/config.sh @@ -5,7 +5,7 @@ OS_NAME=centos OS_VERSION=${OS_VERSION:-stream9} -PACKAGES=chrony +PACKAGES="chrony iptables" [ "$AGENT_INIT" = no ] && PACKAGES+=" systemd" [ "$SECCOMP" = yes ] && PACKAGES+=" libseccomp" diff --git a/tools/osbuilder/rootfs-builder/clearlinux/config.sh b/tools/osbuilder/rootfs-builder/clearlinux/config.sh index cf8497624..e447407ac 100644 --- a/tools/osbuilder/rootfs-builder/clearlinux/config.sh +++ b/tools/osbuilder/rootfs-builder/clearlinux/config.sh @@ -15,12 +15,12 @@ clr_url="https://download.clearlinux.org" BASE_URL="${clr_url}/releases/${OS_VERSION}/${REPO_NAME}/${ARCH}/os/" -PACKAGES="libudev0-shim kmod-bin" +PACKAGES="libudev0-shim kmod-bin iptables-bin" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" = "no" ] && PACKAGES+=" systemd chrony iptables-bin util-linux-bin" || true +[ "$AGENT_INIT" = "no" ] && PACKAGES+=" systemd chrony util-linux-bin" || true # Init process must be one of {systemd,kata-agent} INIT_PROCESS=systemd diff --git a/tools/osbuilder/rootfs-builder/debian/config.sh b/tools/osbuilder/rootfs-builder/debian/config.sh index f87370921..203cad9f9 100644 --- a/tools/osbuilder/rootfs-builder/debian/config.sh +++ b/tools/osbuilder/rootfs-builder/debian/config.sh @@ -8,7 +8,7 @@ OS_VERSION=${OS_VERSION:-10.11} # Set OS_NAME to the desired debian "codename" OS_NAME=${OS_NAME:-"stretch"} -PACKAGES="systemd coreutils init chrony kmod" +PACKAGES="systemd coreutils init iptables chrony kmod" # NOTE: Re-using ubuntu rootfs configuration, see 'ubuntu' folder for full content. source $script_dir/ubuntu/$CONFIG_SH diff --git a/tools/osbuilder/rootfs-builder/ubuntu/config.sh b/tools/osbuilder/rootfs-builder/ubuntu/config.sh index 8afc34804..97a7353e8 100644 --- a/tools/osbuilder/rootfs-builder/ubuntu/config.sh +++ b/tools/osbuilder/rootfs-builder/ubuntu/config.sh @@ -5,7 +5,7 @@ OS_NAME=ubuntu # This should be Ubuntu's code name, e.g. "focal" (Focal Fossa) for 20.04 OS_VERSION=${OS_VERSION:-focal} -PACKAGES=chrony +PACKAGES="chrony iptables" [ "$AGENT_INIT" = no ] && PACKAGES+=" init" [ "$SECCOMP" = yes ] && PACKAGES+=" libseccomp2" REPO_URL=http://ports.ubuntu.com From bd50d463b245a0c7841ff384d1873b437f0c5a4d Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Thu, 21 Apr 2022 21:36:36 -0700 Subject: [PATCH 13/35] agent: iptables: get/set handling for iptables Initial support for getting and setting iptables in the guest. Signed-off-by: Eric Ernst --- src/agent/src/rpc.rs | 311 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 308 insertions(+), 3 deletions(-) diff --git a/src/agent/src/rpc.rs b/src/agent/src/rpc.rs index ebda9e945..bcf2096d2 100644 --- a/src/agent/src/rpc.rs +++ b/src/agent/src/rpc.rs @@ -23,8 +23,9 @@ use cgroups::freezer::FreezerState; use oci::{LinuxNamespace, Root, Spec}; use protobuf::{Message, RepeatedField, SingularPtrField}; use protocols::agent::{ - AddSwapRequest, AgentDetails, CopyFileRequest, GuestDetailsResponse, Interfaces, Metrics, - OOMEvent, ReadStreamResponse, Routes, StatsContainerResponse, VolumeStatsRequest, + AddSwapRequest, AgentDetails, CopyFileRequest, GetIPTablesRequest, GetIPTablesResponse, + GuestDetailsResponse, Interfaces, Metrics, OOMEvent, ReadStreamResponse, Routes, + SetIPTablesRequest, SetIPTablesResponse, StatsContainerResponse, VolumeStatsRequest, WaitProcessResponse, WriteStreamResponse, }; use protocols::csi::{VolumeCondition, VolumeStatsResponse, VolumeUsage, VolumeUsage_Unit}; @@ -75,18 +76,29 @@ use std::time::Duration; use nix::unistd::{Gid, Uid}; use std::fs::{File, OpenOptions}; -use std::io::{BufRead, BufReader}; +use std::io::{BufRead, BufReader, Write}; use std::os::unix::fs::FileExt; use std::path::PathBuf; const CONTAINER_BASE: &str = "/run/kata-containers"; const MODPROBE_PATH: &str = "/sbin/modprobe"; +const IPTABLES_SAVE: &str = "/sbin/iptables-save"; +const IPTABLES_RESTORE: &str = "/sbin/iptables-restore"; +const IP6TABLES_SAVE: &str = "/sbin/ip6tables-save"; +const IP6TABLES_RESTORE: &str = "/sbin/ip6tables-restore"; + const ERR_CANNOT_GET_WRITER: &str = "Cannot get writer"; const ERR_INVALID_BLOCK_SIZE: &str = "Invalid block size"; const ERR_NO_LINUX_FIELD: &str = "Spec does not contain linux field"; const ERR_NO_SANDBOX_PIDNS: &str = "Sandbox does not have sandbox_pidns"; +// IPTABLES_RESTORE_WAIT_SEC is the timeout value provided to iptables-restore --wait. Since we +// don't expect other writers to iptables, we don't expect contention for grabbing the iptables +// filesystem lock. Based on this, 5 seconds seems a resonable timeout period in case the lock is +// not available. +const IPTABLES_RESTORE_WAIT_SEC: u64 = 5; + // Convenience macro to obtain the scope logger macro_rules! sl { () => { @@ -994,6 +1006,140 @@ impl protocols::agent_ttrpc::AgentService for AgentService { }) } + async fn get_ip_tables( + &self, + ctx: &TtrpcContext, + req: GetIPTablesRequest, + ) -> ttrpc::Result { + trace_rpc_call!(ctx, "get_iptables", req); + is_allowed!(req); + + info!(sl!(), "get_ip_tables: request received"); + + let cmd = if req.is_ipv6 { + IP6TABLES_SAVE + } else { + IPTABLES_SAVE + } + .to_string(); + + match Command::new(cmd.clone()).output() { + Ok(output) => Ok(GetIPTablesResponse { + data: output.stdout, + ..Default::default() + }), + Err(e) => { + warn!(sl!(), "failed to run {}: {:?}", cmd, e.kind()); + return Err(ttrpc_error!(ttrpc::Code::INTERNAL, e)); + } + } + } + + async fn set_ip_tables( + &self, + ctx: &TtrpcContext, + req: SetIPTablesRequest, + ) -> ttrpc::Result { + trace_rpc_call!(ctx, "set_iptables", req); + is_allowed!(req); + + info!(sl!(), "set_ip_tables request received"); + + let cmd = if req.is_ipv6 { + IP6TABLES_RESTORE + } else { + IPTABLES_RESTORE + } + .to_string(); + + let mut child = match Command::new(cmd.clone()) + .arg("--wait") + .arg(IPTABLES_RESTORE_WAIT_SEC.to_string()) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + { + Ok(child) => child, + Err(e) => { + warn!(sl!(), "failure to spawn {}: {:?}", cmd, e.kind()); + return Err(ttrpc_error!(ttrpc::Code::INTERNAL, e)); + } + }; + + let mut stdin = match child.stdin.take() { + Some(si) => si, + None => { + println!("failed to get stdin from child"); + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + "failed to take stdin from child".to_string() + )); + } + }; + + let (tx, rx) = tokio::sync::oneshot::channel::(); + let handle = tokio::spawn(async move { + let _ = match stdin.write_all(&req.data) { + Ok(o) => o, + Err(e) => { + warn!(sl!(), "error writing stdin: {:?}", e.kind()); + return; + } + }; + if tx.send(1).is_err() { + warn!(sl!(), "stdin writer thread receiver dropped"); + }; + }); + + if tokio::time::timeout(Duration::from_secs(IPTABLES_RESTORE_WAIT_SEC), rx) + .await + .is_err() + { + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + "timeout waiting for stdin writer to complete".to_string() + )); + } + + if handle.await.is_err() { + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + "stdin writer thread failure".to_string() + )); + } + + let output = match child.wait_with_output() { + Ok(o) => o, + Err(e) => { + warn!( + sl!(), + "failure waiting for spawned {} to complete: {:?}", + cmd, + e.kind() + ); + return Err(ttrpc_error!(ttrpc::Code::INTERNAL, e)); + } + }; + + if !output.status.success() { + warn!(sl!(), "{} failed: {:?}", cmd, output.stderr); + return Err(ttrpc_error!( + ttrpc::Code::INTERNAL, + format!( + "{} failed: {:?}", + cmd, + String::from_utf8_lossy(&output.stderr) + ) + )); + } + + Ok(SetIPTablesResponse { + data: output.stdout, + ..Default::default() + }) + } + async fn list_interfaces( &self, ctx: &TtrpcContext, @@ -1879,6 +2025,7 @@ mod tests { skip_if_not_root, }; use nix::mount; + use nix::sched::{unshare, CloneFlags}; use oci::{Hook, Hooks, Linux, LinuxNamespace}; use tempfile::{tempdir, TempDir}; use ttrpc::{r#async::TtrpcContext, MessageHeader}; @@ -2815,4 +2962,162 @@ OtherField:other assert_eq!(stats.used, 3); assert_eq!(stats.available, available - 2); } + + #[tokio::test] + async fn test_ip_tables() { + skip_if_not_root!(); + + let logger = slog::Logger::root(slog::Discard, o!()); + let sandbox = Sandbox::new(&logger).unwrap(); + let agent_service = Box::new(AgentService { + sandbox: Arc::new(Mutex::new(sandbox)), + }); + + let ctx = mk_ttrpc_context(); + + // Move to a new netns in order to ensure we don't trash the hosts' iptables + unshare(CloneFlags::CLONE_NEWNET).unwrap(); + + // Get initial iptables, we expect to be empty: + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: false, + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "get ip tables should succeed"); + assert_eq!( + result.unwrap().data.len(), + 0, + "ip tables should be empty initially" + ); + + // Initial ip6 ip tables should also be empty: + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: true, + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "get ip6 tables should succeed"); + assert_eq!( + result.unwrap().data.len(), + 0, + "ip tables should be empty initially" + ); + + // Verify that attempting to write 'empty' iptables results in no error: + let empty_rules = ""; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: false, + data: empty_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "set ip tables with no data should succeed"); + + // Verify that attempting to write "garbage" iptables results in an error: + let garbage_rules = r#" +this +is +just garbage +"#; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: false, + data: garbage_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_err(), "set iptables with garbage should fail"); + + // Verify setup of valid iptables:Setup valid set of iptables: + let valid_rules = r#" +*nat +-A PREROUTING -d 192.168.103.153/32 -j DNAT --to-destination 192.168.188.153 + +COMMIT + +"#; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: false, + data: valid_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "set ip tables should succeed"); + + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: false, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!(!result.data.is_empty(), "we should have non-zero output:"); + assert!( + std::str::from_utf8(&*result.data).unwrap().contains( + "PREROUTING -d 192.168.103.153/32 -j DNAT --to-destination 192.168.188.153" + ), + "We should see the resulting rule" + ); + + // Verify setup of valid ip6tables: + let valid_ipv6_rules = r#" +*filter +-A INPUT -s 2001:db8:100::1/128 -i sit+ -p tcp -m tcp --sport 512:65535 + +COMMIT + +"#; + let result = agent_service + .set_ip_tables( + &ctx, + SetIPTablesRequest { + is_ipv6: true, + data: valid_ipv6_rules.as_bytes().to_vec(), + ..Default::default() + }, + ) + .await; + assert!(result.is_ok(), "set ip6 tables should succeed"); + + let result = agent_service + .get_ip_tables( + &ctx, + GetIPTablesRequest { + is_ipv6: true, + ..Default::default() + }, + ) + .await + .unwrap(); + assert!(!result.data.is_empty(), "we should have non-zero output:"); + assert!( + std::str::from_utf8(&*result.data) + .unwrap() + .contains("INPUT -s 2001:db8:100::1/128 -i sit+ -p tcp -m tcp --sport 512:65535"), + "We should see the resulting rule" + ); + } } From 0136be22ca36c5feaded744467ddd58929c6984d Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Mon, 25 Apr 2022 05:07:14 -0700 Subject: [PATCH 14/35] virtcontainers: plumb iptable set/get from sandbox to agent Introduce get/set iptable handling. We add a sandbox API for getting and setting the IPTables within the guest. This routes it from sandbox interface, through kata-agent, ultimately making requests to the guest agent. Signed-off-by: Eric Ernst --- src/runtime/virtcontainers/agent.go | 8 +++++- src/runtime/virtcontainers/interfaces.go | 3 ++ src/runtime/virtcontainers/kata_agent.go | 28 +++++++++++++++++++ src/runtime/virtcontainers/mock_agent.go | 8 ++++++ src/runtime/virtcontainers/pkg/mock/mock.go | 13 +++++++-- .../virtcontainers/pkg/vcmock/sandbox.go | 8 ++++++ src/runtime/virtcontainers/sandbox.go | 10 +++++++ 7 files changed, 75 insertions(+), 3 deletions(-) diff --git a/src/runtime/virtcontainers/agent.go b/src/runtime/virtcontainers/agent.go index 0db392768..90730153e 100644 --- a/src/runtime/virtcontainers/agent.go +++ b/src/runtime/virtcontainers/agent.go @@ -190,9 +190,15 @@ type agent interface { // getAgentMetrics get metrics of agent and guest through agent getAgentMetrics(context.Context, *grpc.GetMetricsRequest) (*grpc.Metrics, error) - //getGuestVolumeStats get the filesystem stats of a volume specified by the volume mount path on the guest. + // getGuestVolumeStats get the filesystem stats of a volume specified by the volume mount path on the guest. getGuestVolumeStats(ctx context.Context, volumeGuestPath string) ([]byte, error) // resizeGuestVolume resizes a volume specified by the volume mount path on the guest. resizeGuestVolume(ctx context.Context, volumeGuestPath string, size uint64) error + + // getIPTables obtains the iptables from the guest + getIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) + + // setIPTables sets the iptables from the guest + setIPTables(ctx context.Context, isIPv6 bool, data []byte) error } diff --git a/src/runtime/virtcontainers/interfaces.go b/src/runtime/virtcontainers/interfaces.go index bb3935be0..df72978e5 100644 --- a/src/runtime/virtcontainers/interfaces.go +++ b/src/runtime/virtcontainers/interfaces.go @@ -79,6 +79,9 @@ type VCSandbox interface { GuestVolumeStats(ctx context.Context, volumePath string) ([]byte, error) ResizeGuestVolume(ctx context.Context, volumePath string, size uint64) error + + GetIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) + SetIPTables(ctx context.Context, isIPv6 bool, data []byte) error } // VCContainer is the Container interface diff --git a/src/runtime/virtcontainers/kata_agent.go b/src/runtime/virtcontainers/kata_agent.go index bee94ef9b..2f85a2828 100644 --- a/src/runtime/virtcontainers/kata_agent.go +++ b/src/runtime/virtcontainers/kata_agent.go @@ -141,6 +141,8 @@ const ( grpcAddSwapRequest = "grpc.AddSwapRequest" grpcVolumeStatsRequest = "grpc.VolumeStatsRequest" grpcResizeVolumeRequest = "grpc.ResizeVolumeRequest" + grpcGetIPTablesRequest = "grpc.GetIPTablesRequest" + grpcSetIPTablesRequest = "grpc.SetIPTablesRequest" ) // newKataAgent returns an agent from an agent type. @@ -1976,6 +1978,12 @@ func (k *kataAgent) installReqFunc(c *kataclient.AgentClient) { k.reqHandlers[grpcResizeVolumeRequest] = func(ctx context.Context, req interface{}) (interface{}, error) { return k.client.AgentServiceClient.ResizeVolume(ctx, req.(*grpc.ResizeVolumeRequest)) } + k.reqHandlers[grpcGetIPTablesRequest] = func(ctx context.Context, req interface{}) (interface{}, error) { + return k.client.AgentServiceClient.GetIPTables(ctx, req.(*grpc.GetIPTablesRequest)) + } + k.reqHandlers[grpcSetIPTablesRequest] = func(ctx context.Context, req interface{}) (interface{}, error) { + return k.client.AgentServiceClient.SetIPTables(ctx, req.(*grpc.SetIPTablesRequest)) + } } func (k *kataAgent) getReqContext(ctx context.Context, reqName string) (newCtx context.Context, cancel context.CancelFunc) { @@ -2194,6 +2202,26 @@ func (k *kataAgent) getAgentMetrics(ctx context.Context, req *grpc.GetMetricsReq return resp.(*grpc.Metrics), nil } +func (k *kataAgent) getIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + resp, err := k.sendReq(ctx, &grpc.GetIPTablesRequest{IsIpv6: isIPv6}) + if err != nil { + return nil, err + } + return resp.(*grpc.GetIPTablesResponse).Data, nil +} + +func (k *kataAgent) setIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + _, err := k.sendReq(ctx, &grpc.SetIPTablesRequest{ + IsIpv6: isIPv6, + Data: data, + }) + if err != nil { + k.Logger().WithError(err).Errorf("setIPTables request to agent failed") + } + + return err +} + func (k *kataAgent) getGuestVolumeStats(ctx context.Context, volumeGuestPath string) ([]byte, error) { result, err := k.sendReq(ctx, &grpc.VolumeStatsRequest{VolumeGuestPath: volumeGuestPath}) if err != nil { diff --git a/src/runtime/virtcontainers/mock_agent.go b/src/runtime/virtcontainers/mock_agent.go index 43c84b947..a3c4d44b0 100644 --- a/src/runtime/virtcontainers/mock_agent.go +++ b/src/runtime/virtcontainers/mock_agent.go @@ -249,3 +249,11 @@ func (n *mockAgent) getGuestVolumeStats(ctx context.Context, volumeGuestPath str func (n *mockAgent) resizeGuestVolume(ctx context.Context, volumeGuestPath string, size uint64) error { return nil } + +func (k *mockAgent) getIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + return nil, nil +} + +func (k *mockAgent) setIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + return nil +} diff --git a/src/runtime/virtcontainers/pkg/mock/mock.go b/src/runtime/virtcontainers/pkg/mock/mock.go index 2af88b841..634dc7fd8 100644 --- a/src/runtime/virtcontainers/pkg/mock/mock.go +++ b/src/runtime/virtcontainers/pkg/mock/mock.go @@ -12,12 +12,13 @@ import ( "net/url" "os" + "path" + "strings" + "github.com/containerd/ttrpc" gpb "github.com/gogo/protobuf/types" aTypes "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/agent/protocols" pb "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/agent/protocols/grpc" - "path" - "strings" ) const VSockPrefix = "mock://" @@ -239,3 +240,11 @@ func (p *HybridVSockTTRPCMockImp) GetVolumeStats(ctx context.Context, req *pb.Vo func (p *HybridVSockTTRPCMockImp) ResizeVolume(ctx context.Context, req *pb.ResizeVolumeRequest) (*gpb.Empty, error) { return &gpb.Empty{}, nil } + +func (p *HybridVSockTTRPCMockImp) GetIPTables(ctx context.Context, req *pb.GetIPTablesRequest) (*pb.GetIPTablesResponse, error) { + return &pb.GetIPTablesResponse{}, nil +} + +func (p *HybridVSockTTRPCMockImp) SetIPTables(ctx context.Context, req *pb.SetIPTablesRequest) (*pb.SetIPTablesResponse, error) { + return &pb.SetIPTablesResponse{}, nil +} diff --git a/src/runtime/virtcontainers/pkg/vcmock/sandbox.go b/src/runtime/virtcontainers/pkg/vcmock/sandbox.go index b3b5f7a8d..064a60af7 100644 --- a/src/runtime/virtcontainers/pkg/vcmock/sandbox.go +++ b/src/runtime/virtcontainers/pkg/vcmock/sandbox.go @@ -261,3 +261,11 @@ func (s *Sandbox) GuestVolumeStats(ctx context.Context, path string) ([]byte, er func (s *Sandbox) ResizeGuestVolume(ctx context.Context, path string, size uint64) error { return nil } + +func (s *Sandbox) GetIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + return nil, nil +} + +func (s *Sandbox) SetIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + return nil +} diff --git a/src/runtime/virtcontainers/sandbox.go b/src/runtime/virtcontainers/sandbox.go index d494d54ba..71851c2cf 100644 --- a/src/runtime/virtcontainers/sandbox.go +++ b/src/runtime/virtcontainers/sandbox.go @@ -2254,6 +2254,16 @@ func (s *Sandbox) GetAgentURL() (string, error) { return s.agent.getAgentURL() } +// GetIPTables will obtain the iptables from the guest +func (s *Sandbox) GetIPTables(ctx context.Context, isIPv6 bool) ([]byte, error) { + return s.agent.getIPTables(ctx, isIPv6) +} + +// SetIPTables will set the iptables in the guest +func (s *Sandbox) SetIPTables(ctx context.Context, isIPv6 bool, data []byte) error { + return s.agent.setIPTables(ctx, isIPv6, data) +} + // GuestVolumeStats return the filesystem stat of a given volume in the guest. func (s *Sandbox) GuestVolumeStats(ctx context.Context, volumePath string) ([]byte, error) { guestMountPath, err := s.guestMountPath(volumePath) From 640173cfc252f41630f3c377d0d996c9e7ab1f01 Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Mon, 25 Apr 2022 04:31:25 -0700 Subject: [PATCH 15/35] shim-mgmt: Add endpoint handler for interacting with iptables Add two endpoints: ip6tables, iptables. Each url handler supports GET and PUT operations. PUT expects the requests' data to be []bytes, and to contain iptable information in format to be consumed by iptables-restore. Signed-off-by: Eric Ernst --- .../pkg/containerd-shim-v2/shim_management.go | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/src/runtime/pkg/containerd-shim-v2/shim_management.go b/src/runtime/pkg/containerd-shim-v2/shim_management.go index e10922250..00eb9d7db 100644 --- a/src/runtime/pkg/containerd-shim-v2/shim_management.go +++ b/src/runtime/pkg/containerd-shim-v2/shim_management.go @@ -29,6 +29,7 @@ import ( "github.com/prometheus/client_golang/prometheus" dto "github.com/prometheus/client_model/go" "github.com/prometheus/common/expfmt" + "github.com/sirupsen/logrus" ) const ( @@ -36,6 +37,8 @@ const ( DirectVolumeStatUrl = "/direct-volume/stats" DirectVolumeResizeUrl = "/direct-volume/resize" + IPTablesUrl = "/iptables" + IP6TablesUrl = "/ip6tables" ) var ( @@ -195,6 +198,48 @@ func (s *service) serveVolumeResize(w http.ResponseWriter, r *http.Request) { w.Write([]byte("")) } +func (s *service) ip6TablesHandler(w http.ResponseWriter, r *http.Request) { + s.genericIPTablesHandler(w, r, true) +} + +func (s *service) ipTablesHandler(w http.ResponseWriter, r *http.Request) { + s.genericIPTablesHandler(w, r, false) +} + +func (s *service) genericIPTablesHandler(w http.ResponseWriter, r *http.Request, isIPv6 bool) { + logger := shimMgtLog.WithFields(logrus.Fields{"handler": "iptables", "ipv6": isIPv6}) + + switch r.Method { + case http.MethodPut: + body, err := ioutil.ReadAll(r.Body) + if err != nil { + logger.WithError(err).Error("failed to read request body") + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(err.Error())) + return + } + + if err = s.sandbox.SetIPTables(context.Background(), isIPv6, body); err != nil { + logger.WithError(err).Error("failed to set IPTables") + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(err.Error())) + } + w.Write([]byte("")) + + case http.MethodGet: + buf, err := s.sandbox.GetIPTables(context.Background(), isIPv6) + if err != nil { + logger.WithError(err).Error("failed to get IPTables") + w.WriteHeader(http.StatusInternalServerError) + w.Write([]byte(err.Error())) + } + w.Write(buf) + default: + w.WriteHeader(http.StatusNotImplemented) + return + } +} + func (s *service) startManagementServer(ctx context.Context, ociSpec *specs.Spec) { // metrics socket will under sandbox's bundle path metricsAddress := SocketAddress(s.id) @@ -219,6 +264,8 @@ func (s *service) startManagementServer(ctx context.Context, ociSpec *specs.Spec m.Handle("/agent-url", http.HandlerFunc(s.agentURL)) m.Handle(DirectVolumeStatUrl, http.HandlerFunc(s.serveVolumeStats)) m.Handle(DirectVolumeResizeUrl, http.HandlerFunc(s.serveVolumeResize)) + m.Handle(IPTablesUrl, http.HandlerFunc(s.ipTablesHandler)) + m.Handle(IP6TablesUrl, http.HandlerFunc(s.ip6TablesHandler)) s.mountPprofHandle(m, ociSpec) // register shim metrics From 2a09378dd90603d0021ed82ab08637a6885942ce Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Mon, 25 Apr 2022 04:31:25 -0700 Subject: [PATCH 16/35] shim-client: add support for DoPut While at it, make sure we check for nil in DoPost Signed-off-by: Eric Ernst --- src/runtime/cmd/kata-runtime/kata-volume.go | 2 +- .../shimclient/shim_management_client.go | 31 +++++++++++++++++-- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/src/runtime/cmd/kata-runtime/kata-volume.go b/src/runtime/cmd/kata-runtime/kata-volume.go index 55274f7d8..b33febe19 100644 --- a/src/runtime/cmd/kata-runtime/kata-volume.go +++ b/src/runtime/cmd/kata-runtime/kata-volume.go @@ -163,5 +163,5 @@ func Resize(volumePath string, size uint64) error { if err != nil { return err } - return shimclient.DoPost(sandboxId, defaultTimeout, containerdshim.DirectVolumeResizeUrl, encoded) + return shimclient.DoPost(sandboxId, defaultTimeout, containerdshim.DirectVolumeResizeUrl, "application/json", encoded) } diff --git a/src/runtime/pkg/utils/shimclient/shim_management_client.go b/src/runtime/pkg/utils/shimclient/shim_management_client.go index c9ed3ad69..835b6f9b9 100644 --- a/src/runtime/pkg/utils/shimclient/shim_management_client.go +++ b/src/runtime/pkg/utils/shimclient/shim_management_client.go @@ -65,15 +65,40 @@ func DoGet(sandboxID string, timeoutInSeconds time.Duration, urlPath string) ([] return body, nil } -func DoPost(sandboxID string, timeoutInSeconds time.Duration, urlPath string, payload []byte) error { +// DoPut will make a PUT request to the shim endpoint that handles the given sandbox ID +func DoPut(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentType string, payload []byte) error { client, err := BuildShimClient(sandboxID, timeoutInSeconds) if err != nil { return err } - resp, err := client.Post(fmt.Sprintf("http://shim/%s", urlPath), "application/json", bytes.NewBuffer(payload)) + req, err := http.NewRequest(http.MethodPut, fmt.Sprintf("http://shim/%s", urlPath), bytes.NewBuffer(payload)) + if err != nil { + return err + } + req.Header.Set("Content-Type", contentType) + + resp, err := client.Do(req) defer func() { - resp.Body.Close() + if resp != nil { + resp.Body.Close() + } + }() + return err +} + +// DoPost will make a POST request to the shim endpoint that handles the given sandbox ID +func DoPost(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentType string, payload []byte) error { + client, err := BuildShimClient(sandboxID, timeoutInSeconds) + if err != nil { + return err + } + + resp, err := client.Post(fmt.Sprintf("http://shim/%s", urlPath), contentType, bytes.NewBuffer(payload)) + defer func() { + if resp != nil { + resp.Body.Close() + } }() return err } From 0706fb28ac14eaa395962c8a33fc9e34f58e0a7f Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Wed, 27 Apr 2022 22:44:17 -0700 Subject: [PATCH 17/35] kata-runtime: shmgmt: make url usage consistent Before, we had a mix of slash, etc. Unfortunately, when cleaning URL paths, serve mux seems to mangle the request method, resulting in each request being a GET (instead of PUT or POST). Signed-off-by: Eric Ernst --- src/runtime/pkg/containerd-shim-v2/shim_management.go | 9 +++++---- src/runtime/pkg/kata-monitor/metrics.go | 5 +++-- .../pkg/utils/shimclient/shim_management_client.go | 6 +++--- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/src/runtime/pkg/containerd-shim-v2/shim_management.go b/src/runtime/pkg/containerd-shim-v2/shim_management.go index 00eb9d7db..a6b9bdba3 100644 --- a/src/runtime/pkg/containerd-shim-v2/shim_management.go +++ b/src/runtime/pkg/containerd-shim-v2/shim_management.go @@ -33,12 +33,13 @@ import ( ) const ( - DirectVolumePathKey = "path" - + DirectVolumePathKey = "path" + AgentUrl = "/agent-url" DirectVolumeStatUrl = "/direct-volume/stats" DirectVolumeResizeUrl = "/direct-volume/resize" IPTablesUrl = "/iptables" IP6TablesUrl = "/ip6tables" + MetricsUrl = "/metrics" ) var ( @@ -260,8 +261,8 @@ func (s *service) startManagementServer(ctx context.Context, ociSpec *specs.Spec // bind handler m := http.NewServeMux() - m.Handle("/metrics", http.HandlerFunc(s.serveMetrics)) - m.Handle("/agent-url", http.HandlerFunc(s.agentURL)) + m.Handle(MetricsUrl, http.HandlerFunc(s.serveMetrics)) + m.Handle(AgentUrl, http.HandlerFunc(s.agentURL)) m.Handle(DirectVolumeStatUrl, http.HandlerFunc(s.serveVolumeStats)) m.Handle(DirectVolumeResizeUrl, http.HandlerFunc(s.serveVolumeResize)) m.Handle(IPTablesUrl, http.HandlerFunc(s.ipTablesHandler)) diff --git a/src/runtime/pkg/kata-monitor/metrics.go b/src/runtime/pkg/kata-monitor/metrics.go index e5d947767..98ecb68f0 100644 --- a/src/runtime/pkg/kata-monitor/metrics.go +++ b/src/runtime/pkg/kata-monitor/metrics.go @@ -15,6 +15,7 @@ import ( "sync" "time" + containerdshim "github.com/kata-containers/kata-containers/src/runtime/pkg/containerd-shim-v2" mutils "github.com/kata-containers/kata-containers/src/runtime/pkg/utils" "github.com/kata-containers/kata-containers/src/runtime/pkg/utils/shimclient" "github.com/prometheus/client_golang/prometheus" @@ -239,7 +240,7 @@ func (km *KataMonitor) aggregateSandboxMetrics(encoder expfmt.Encoder) error { } func getParsedMetrics(sandboxID string, sandboxMetadata sandboxCRIMetadata) ([]*dto.MetricFamily, error) { - body, err := shimclient.DoGet(sandboxID, defaultTimeout, "metrics") + body, err := shimclient.DoGet(sandboxID, defaultTimeout, containerdshim.MetricsUrl) if err != nil { return nil, err } @@ -249,7 +250,7 @@ func getParsedMetrics(sandboxID string, sandboxMetadata sandboxCRIMetadata) ([]* // GetSandboxMetrics will get sandbox's metrics from shim func GetSandboxMetrics(sandboxID string) (string, error) { - body, err := shimclient.DoGet(sandboxID, defaultTimeout, "metrics") + body, err := shimclient.DoGet(sandboxID, defaultTimeout, containerdshim.MetricsUrl) if err != nil { return "", err } diff --git a/src/runtime/pkg/utils/shimclient/shim_management_client.go b/src/runtime/pkg/utils/shimclient/shim_management_client.go index 835b6f9b9..81c0d70de 100644 --- a/src/runtime/pkg/utils/shimclient/shim_management_client.go +++ b/src/runtime/pkg/utils/shimclient/shim_management_client.go @@ -48,7 +48,7 @@ func DoGet(sandboxID string, timeoutInSeconds time.Duration, urlPath string) ([] return nil, err } - resp, err := client.Get(fmt.Sprintf("http://shim/%s", urlPath)) + resp, err := client.Get(fmt.Sprintf("http://shim%s", urlPath)) if err != nil { return nil, err } @@ -72,7 +72,7 @@ func DoPut(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentTyp return err } - req, err := http.NewRequest(http.MethodPut, fmt.Sprintf("http://shim/%s", urlPath), bytes.NewBuffer(payload)) + req, err := http.NewRequest(http.MethodPut, fmt.Sprintf("http://shim%s", urlPath), bytes.NewBuffer(payload)) if err != nil { return err } @@ -94,7 +94,7 @@ func DoPost(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentTy return err } - resp, err := client.Post(fmt.Sprintf("http://shim/%s", urlPath), contentType, bytes.NewBuffer(payload)) + resp, err := client.Post(fmt.Sprintf("http://shim%s", urlPath), contentType, bytes.NewBuffer(payload)) defer func() { if resp != nil { resp.Body.Close() From 3201ad0830c857f98ea6549fccd3d9b4be5e979c Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Wed, 25 May 2022 10:10:28 -0700 Subject: [PATCH 18/35] shim-client: ensure we check resp status for Put/Post Without this, potential errors are silently dropped. Let's ensure we return the error code as well as potenial data from the response. Signed-off-by: Eric Ernst --- .../shimclient/shim_management_client.go | 25 +++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/src/runtime/pkg/utils/shimclient/shim_management_client.go b/src/runtime/pkg/utils/shimclient/shim_management_client.go index 81c0d70de..3f9e68650 100644 --- a/src/runtime/pkg/utils/shimclient/shim_management_client.go +++ b/src/runtime/pkg/utils/shimclient/shim_management_client.go @@ -9,6 +9,7 @@ import ( "bytes" "fmt" "io" + "io/ioutil" "net" "net/http" "time" @@ -79,12 +80,22 @@ func DoPut(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentTyp req.Header.Set("Content-Type", contentType) resp, err := client.Do(req) + if err != nil { + return err + } + defer func() { if resp != nil { resp.Body.Close() } }() - return err + + if resp.StatusCode != http.StatusOK { + data, _ := ioutil.ReadAll(resp.Body) + return fmt.Errorf("error sending put: url: %s, status code: %d, response data: %s", urlPath, resp.StatusCode, string(data)) + } + + return nil } // DoPost will make a POST request to the shim endpoint that handles the given sandbox ID @@ -95,10 +106,20 @@ func DoPost(sandboxID string, timeoutInSeconds time.Duration, urlPath, contentTy } resp, err := client.Post(fmt.Sprintf("http://shim%s", urlPath), contentType, bytes.NewBuffer(payload)) + if err != nil { + return err + } + defer func() { if resp != nil { resp.Body.Close() } }() - return err + + if resp.StatusCode != http.StatusOK { + data, _ := ioutil.ReadAll(resp.Body) + return fmt.Errorf("error sending post: url: %s, status code: %d, response data: %s", urlPath, resp.StatusCode, string(data)) + } + + return nil } From 65f0cef16c77b9786fa4a64e12a54ac18d2d7f6f Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Mon, 25 Apr 2022 04:17:47 -0700 Subject: [PATCH 19/35] kata-runtime: add iptables CLI to test http endpoint While end users can connect directly to the shim, let's provide a way to easily get/set iptables from kata-runtime itself. Fixes: #4080 Signed-off-by: Eric Ernst --- src/runtime/cmd/kata-runtime/kata-iptables.go | 122 ++++++++++++++++++ src/runtime/cmd/kata-runtime/main.go | 1 + 2 files changed, 123 insertions(+) create mode 100644 src/runtime/cmd/kata-runtime/kata-iptables.go diff --git a/src/runtime/cmd/kata-runtime/kata-iptables.go b/src/runtime/cmd/kata-runtime/kata-iptables.go new file mode 100644 index 000000000..8e2bd93c6 --- /dev/null +++ b/src/runtime/cmd/kata-runtime/kata-iptables.go @@ -0,0 +1,122 @@ +// Copyright (c) 2022 Apple Inc. +// +// SPDX-License-Identifier: Apache-2.0 +// + +package main + +import ( + "fmt" + "io/ioutil" + + containerdshim "github.com/kata-containers/kata-containers/src/runtime/pkg/containerd-shim-v2" + "github.com/kata-containers/kata-containers/src/runtime/pkg/katautils" + "github.com/kata-containers/kata-containers/src/runtime/pkg/utils/shimclient" + "github.com/urfave/cli" +) + +var ( + sandboxID string + isIPv6 bool +) +var iptablesSubCmds = []cli.Command{ + getIPTablesCommand, + setIPTablesCommand, +} + +var kataIPTablesCommand = cli.Command{ + Name: "iptables", + Usage: "get or set iptables within the Kata Containers guest", + Subcommands: iptablesSubCmds, + Action: func(context *cli.Context) { + cli.ShowSubcommandHelp(context) + }, +} + +var getIPTablesCommand = cli.Command{ + Name: "get", + Usage: "get iptables from the Kata Containers guest", + Flags: []cli.Flag{ + cli.StringFlag{ + Name: "sandbox-id", + Usage: "the target sandbox for getting the iptables", + Required: true, + Destination: &sandboxID, + }, + cli.BoolFlag{ + Name: "v6", + Usage: "indicate we're requesting ipv6 iptables", + Destination: &isIPv6, + }, + }, + Action: func(c *cli.Context) error { + // verify sandbox exists: + if err := katautils.VerifyContainerID(sandboxID); err != nil { + return err + } + + url := containerdshim.IPTablesUrl + if isIPv6 { + url = containerdshim.IP6TablesUrl + } + body, err := shimclient.DoGet(sandboxID, defaultTimeout, url) + if err != nil { + return err + } + + fmt.Println(string(body)) + return nil + }, +} + +var setIPTablesCommand = cli.Command{ + Name: "set", + Usage: "set iptables in a specifc Kata Containers guest based on file", + Flags: []cli.Flag{ + cli.StringFlag{ + Name: "sandbox-id", + Usage: "the target sandbox for setting the iptables", + Required: true, + Destination: &sandboxID, + }, + cli.BoolFlag{ + Name: "v6", + Usage: "indicate we're requesting ipv6 iptables", + Destination: &isIPv6, + }, + }, + Action: func(c *cli.Context) error { + iptablesFile := c.Args().Get(0) + + // verify sandbox exists: + if err := katautils.VerifyContainerID(sandboxID); err != nil { + return err + } + + // verify iptables were provided: + if iptablesFile == "" { + return fmt.Errorf("iptables file not provided") + } + + if !katautils.FileExists(iptablesFile) { + return fmt.Errorf("iptables file does not exist: %s", iptablesFile) + } + + // Read file into buffer, and make request to the appropriate shim + buf, err := ioutil.ReadFile(iptablesFile) + if err != nil { + return err + } + + url := containerdshim.IPTablesUrl + if isIPv6 { + url = containerdshim.IP6TablesUrl + } + + if err = shimclient.DoPut(sandboxID, defaultTimeout, url, "application/octet-stream", buf); err != nil { + return fmt.Errorf("Error observed when making iptables-set request(%s): %s", iptablesFile, err) + } + + return nil + }, +} diff --git a/src/runtime/cmd/kata-runtime/main.go b/src/runtime/cmd/kata-runtime/main.go index def7431f0..a722afd6b 100644 --- a/src/runtime/cmd/kata-runtime/main.go +++ b/src/runtime/cmd/kata-runtime/main.go @@ -125,6 +125,7 @@ var runtimeCommands = []cli.Command{ kataMetricsCLICommand, factoryCLICommand, kataVolumeCommand, + kataIPTablesCommand, } // runtimeBeforeSubcommands is the function to run before command-line From af2ef3f7a5e55c121fb7362ed410506089dee6ab Mon Sep 17 00:00:00 2001 From: Eric Ernst Date: Tue, 10 May 2022 14:59:22 -0700 Subject: [PATCH 20/35] agent-ctl: introduce handle for iptables get/set Add support for the updated agent API for iptables Signed-off-by: Eric Ernst --- src/tools/agent-ctl/src/client.rs | 56 +++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/src/tools/agent-ctl/src/client.rs b/src/tools/agent-ctl/src/client.rs index bcaa87844..315969b87 100644 --- a/src/tools/agent-ctl/src/client.rs +++ b/src/tools/agent-ctl/src/client.rs @@ -161,6 +161,11 @@ static AGENT_CMDS: &[AgentCmd] = &[ st: ServiceType::Agent, fp: agent_cmd_sandbox_get_guest_details, }, + AgentCmd { + name: "GetIptables", + st: ServiceType::Agent, + fp: agent_cmd_sandbox_get_ip_tables, + }, AgentCmd { name: "GetMetrics", st: ServiceType::Agent, @@ -231,6 +236,11 @@ static AGENT_CMDS: &[AgentCmd] = &[ st: ServiceType::Agent, fp: agent_cmd_sandbox_set_guest_date_time, }, + AgentCmd { + name: "SetIptables", + st: ServiceType::Agent, + fp: agent_cmd_sandbox_set_ip_tables, + }, AgentCmd { name: "SignalProcess", st: ServiceType::Agent, @@ -1219,6 +1229,29 @@ fn agent_cmd_sandbox_get_guest_details( Ok(()) } +fn agent_cmd_sandbox_get_ip_tables( + ctx: &Context, + client: &AgentServiceClient, + _health: &HealthClient, + _options: &mut Options, + args: &str, +) -> Result<()> { + let req: GetIPTablesRequest = utils::make_request(args)?; + + let ctx = clone_context(ctx); + + debug!(sl!(), "sending request"; "request" => format!("{:?}", req)); + + let reply = client + .get_ip_tables(ctx, &req) + .map_err(|e| anyhow!("{:?}", e).context(ERR_API_FAILED))?; + + info!(sl!(), "response received"; + "response" => format!("{:?}", reply)); + + Ok(()) +} + fn agent_cmd_container_wait_process( ctx: &Context, client: &AgentServiceClient, @@ -1899,6 +1932,29 @@ fn agent_cmd_sandbox_set_guest_date_time( Ok(()) } +fn agent_cmd_sandbox_set_ip_tables( + ctx: &Context, + client: &AgentServiceClient, + _health: &HealthClient, + _options: &mut Options, + args: &str, +) -> Result<()> { + let req: SetIPTablesRequest = utils::make_request(args)?; + + let ctx = clone_context(ctx); + + debug!(sl!(), "sending request"; "request" => format!("{:?}", req)); + + let reply = client + .set_ip_tables(ctx, &req) + .map_err(|e| anyhow!(e).context(ERR_API_FAILED))?; + + info!(sl!(), "response received"; + "response" => format!("{:?}", reply)); + + Ok(()) +} + fn agent_cmd_sandbox_add_arp_neighbors( ctx: &Context, client: &AgentServiceClient, From 5903815746d304d360129890961d07c1b0fc6ce1 Mon Sep 17 00:00:00 2001 From: Manabu Sugimoto Date: Fri, 27 May 2022 15:50:02 +0900 Subject: [PATCH 21/35] agent: Pass standard I/O to container launched by runk The `kata-agent` passes its standard I/O file descriptors through to the container process that will be launched by `runk` without manipulation or modification in order to allow the container process can handle its I/O operations. Fixes: #4327 Signed-off-by: Manabu Sugimoto --- src/agent/rustjail/src/process.rs | 34 +++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/src/agent/rustjail/src/process.rs b/src/agent/rustjail/src/process.rs index cced9b98f..d94b595cc 100644 --- a/src/agent/rustjail/src/process.rs +++ b/src/agent/rustjail/src/process.rs @@ -5,7 +5,7 @@ use libc::pid_t; use std::fs::File; -use std::os::unix::io::RawFd; +use std::os::unix::io::{AsRawFd, RawFd}; use tokio::sync::mpsc::Sender; use nix::errno::Errno; @@ -137,19 +137,25 @@ impl Process { info!(logger, "before create console socket!"); if !p.tty { - info!(logger, "created console socket!"); + if cfg!(feature = "standard-oci-runtime") { + p.stdin = Some(std::io::stdin().as_raw_fd()); + p.stdout = Some(std::io::stdout().as_raw_fd()); + p.stderr = Some(std::io::stderr().as_raw_fd()); + } else { + info!(logger, "created console socket!"); - let (stdin, pstdin) = unistd::pipe2(OFlag::O_CLOEXEC)?; - p.parent_stdin = Some(pstdin); - p.stdin = Some(stdin); + let (stdin, pstdin) = unistd::pipe2(OFlag::O_CLOEXEC)?; + p.parent_stdin = Some(pstdin); + p.stdin = Some(stdin); - let (pstdout, stdout) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; - p.parent_stdout = Some(pstdout); - p.stdout = Some(stdout); + let (pstdout, stdout) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; + p.parent_stdout = Some(pstdout); + p.stdout = Some(stdout); - let (pstderr, stderr) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; - p.parent_stderr = Some(pstderr); - p.stderr = Some(stderr); + let (pstderr, stderr) = create_extended_pipe(OFlag::O_CLOEXEC, pipe_size)?; + p.parent_stderr = Some(pstderr); + p.stderr = Some(stderr); + } } Ok(p) } @@ -284,5 +290,11 @@ mod tests { // group of the calling process. process.pid = 0; assert!(process.signal(libc::SIGCONT).is_ok()); + + if cfg!(feature = "standard-oci-runtime") { + assert_eq!(process.stdin.unwrap(), std::io::stdin().as_raw_fd()); + assert_eq!(process.stdout.unwrap(), std::io::stdout().as_raw_fd()); + assert_eq!(process.stderr.unwrap(), std::io::stderr().as_raw_fd()); + } } } From d759f6c3e5e3324136d34c087068dcf88abd6cde Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Wed, 1 Jun 2022 11:56:43 +0100 Subject: [PATCH 22/35] snap: Fix CH architecture check Correct the `cloud-hypervisor` part architecture check to use `x86_64`, not `x64_64`. Signed-off-by: James O. D. Hunt --- snap/snapcraft.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml index 94fd2a8c9..fd0101db5 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -357,7 +357,7 @@ parts: after: [godeps] override-build: | arch=$(uname -m) - if [ "{$arch}" == "aarch64" ] || [ "${arch}" == "x64_64" ]; then + if [ "${arch}" == "aarch64" ] || [ "${arch}" == "x86_64" ]; then sudo apt-get -y update sudo apt-get -y install ca-certificates curl gnupg lsb-release curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg From 894f661cc40006fbccb34f0cee54038eacf28faf Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Thu, 26 May 2022 14:11:40 +0100 Subject: [PATCH 23/35] docs: Add warning to snap build Since we must build with `--destructive-mode`, add a warning that the host environment could change the behaviour of the build, depending on the packages installed on the system. Signed-off-by: James O. D. Hunt --- snap/local/README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/snap/local/README.md b/snap/local/README.md index 60564f4b2..83d33847f 100644 --- a/snap/local/README.md +++ b/snap/local/README.md @@ -28,6 +28,20 @@ Run the command below which will use the packaging Makefile to build the snap im $ make -C tools/packaging snap ``` +> **Warning:** +> +> By default, `snapcraft` will create a clean virtual machine +> environment to build the snap in using the `multipass` tool. +> +> However, `multipass` is silently disabled when `--destructive-mode` is +> used. +> +> Since building the Kata Containers package currently requires +> `--destructive-mode`, the snap will be built using the host +> environment. To avoid parts of the build auto-detecting additional +> features to enable (for example for QEMU), we recommend that you +> only run the snap build in a minimal host environment. + To install the resulting snap image, snap must be put in [classic mode][3] and the security confinement must be disabled (*--classic*). Also since the resulting snap has not been signed the verification of signature must be omitted (*--dangerous*). From 9b108d9937952b882dc91f397771ffa33f6f392e Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Thu, 26 May 2022 14:12:35 +0100 Subject: [PATCH 24/35] docs: Improve snap formatting Improve the snap docs by using more consistent formatting and proper shell code in the shell example. Signed-off-by: James O. D. Hunt --- snap/local/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/snap/local/README.md b/snap/local/README.md index 83d33847f..47174d174 100644 --- a/snap/local/README.md +++ b/snap/local/README.md @@ -43,14 +43,14 @@ $ make -C tools/packaging snap > only run the snap build in a minimal host environment. To install the resulting snap image, snap must be put in [classic mode][3] and the -security confinement must be disabled (*--classic*). Also since the resulting snap -has not been signed the verification of signature must be omitted (*--dangerous*). +security confinement must be disabled (`--classic`). Also since the resulting snap +has not been signed the verification of signature must be omitted (`--dangerous`). ```sh -$ sudo snap install --classic --dangerous kata-containers_[VERSION]_[ARCH].snap +$ sudo snap install --classic --dangerous "kata-containers_${version}_${arch}.snap" ``` -Replace `VERSION` with the current version of Kata Containers and `ARCH` with +Replace `${version}` with the current version of Kata Containers and `${arch}` with the system architecture. ## Configure Kata Containers From 6d0ff901abac317a90b5e24f5836c0eeff160a28 Mon Sep 17 00:00:00 2001 From: Zvonko Kaiser Date: Mon, 30 May 2022 13:07:21 -0700 Subject: [PATCH 25/35] docs: Update vGPU use-case Now that #4213 is merged we need updated documentation for vGPU time-sliced or vGPU MIG-backed. Fixes: #4343 Signed-off-by: Zvonko Kaiser --- .../NVIDIA-GPU-passthrough-and-Kata.md | 256 ++++++++++++++++-- 1 file changed, 238 insertions(+), 18 deletions(-) diff --git a/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md b/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md index 32943b8d3..18a75bde7 100644 --- a/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md +++ b/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata.md @@ -2,20 +2,20 @@ An NVIDIA GPU device can be passed to a Kata Containers container using GPU passthrough (NVIDIA GPU pass-through mode) as well as GPU mediated passthrough -(NVIDIA vGPU mode). +(NVIDIA `vGPU` mode). NVIDIA GPU pass-through mode, an entire physical GPU is directly assigned to one VM, bypassing the NVIDIA Virtual GPU Manager. In this mode of operation, the GPU is accessed exclusively by the NVIDIA driver running in the VM to which it is assigned. The GPU is not shared among VMs. -NVIDIA Virtual GPU (vGPU) enables multiple virtual machines (VMs) to have +NVIDIA Virtual GPU (`vGPU`) enables multiple virtual machines (VMs) to have simultaneous, direct access to a single physical GPU, using the same NVIDIA graphics drivers that are deployed on non-virtualized operating systems. By -doing this, NVIDIA vGPU provides VMs with unparalleled graphics performance, +doing this, NVIDIA `vGPU` provides VMs with unparalleled graphics performance, compute performance, and application compatibility, together with the cost-effectiveness and scalability brought about by sharing a GPU among multiple -workloads. A vGPU can be either time-sliced or Multi-Instance GPU (MIG)-backed +workloads. A `vGPU` can be either time-sliced or Multi-Instance GPU (MIG)-backed with [MIG-slices](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/). | Technology | Description | Behavior | Detail | @@ -46,14 +46,14 @@ $ lspci -s d0:00.0 -vv | grep Region For large BARs devices, MMIO mapping above 4G address space should be `enabled` in the PCI configuration of the BIOS. -Some hardware vendors use different name in BIOS, such as: +Some hardware vendors use a different name in BIOS, such as: - Above 4G Decoding - Memory Hole for PCI MMIO - Memory Mapped I/O above 4GB If one is using a GPU based on the Ampere architecture and later additionally -SR-IOV needs to be enabled for the vGPU use-case. +SR-IOV needs to be enabled for the `vGPU` use-case. The following steps outline the workflow for using an NVIDIA GPU with Kata. @@ -154,7 +154,7 @@ $ ./build-kernel.sh -v 5.15.23 -g nvidia build $ sudo -E ./build-kernel.sh -v 5.15.23 -g nvidia install ``` -To build NVIDIA Driver in Kata container, `linux-headers` is required. +To build NVIDIA Driver in Kata container, `linux-headers` are required. This is a way to generate deb packages for `linux-headers`: > **Note**: @@ -177,7 +177,7 @@ kernel = "/usr/share/kata-containers/vmlinuz-nvidia-gpu.container" Use the following steps to pass an NVIDIA GPU device in pass-through mode with Kata: -1. Find the Bus-Device-Function (BDF) for GPU device on host: +1. Find the Bus-Device-Function (BDF) for the GPU device on the host: ```sh $ sudo lspci -nn -D | grep -i nvidia @@ -219,7 +219,7 @@ Use the following steps to pass an NVIDIA GPU device in pass-through mode with K crw-rw-rw- 1 root root 10, 196 Mar 18 02:27 vfio ``` -4. Start a Kata container with GPU device: +4. Start a Kata container with the GPU device: ```sh # You may need to `modprobe vhost-vsock` if you get @@ -246,9 +246,228 @@ Use the following steps to pass an NVIDIA GPU device in pass-through mode with K ## NVIDIA vGPU mode with Kata Containers NVIDIA vGPU is a licensed product on all supported GPU boards. A software license -is required to enable all vGPU features within the guest VM. +is required to enable all vGPU features within the guest VM. NVIDIA vGPU manager +needs to be installed on the host to configure GPUs in vGPU mode. See [NVIDIA Virtual GPU Software Documentation v14.0 through 14.1](https://docs.nvidia.com/grid/14.0/) for more details. -> **TODO**: Will follow up with instructions +### NVIDIA vGPU time-sliced + +In the time-sliced mode, the GPU is not partitioned and the workload uses the +whole GPU and shares access to the GPU engines. Processes are scheduled in +series. The best effort scheduler is the default one and can be exchanged by +other scheduling policies see the documentation above how to do that. + +Beware if you had `MIG` enabled before to disable `MIG` on the GPU if you want +to use `time-sliced` `vGPU`. + +```sh +$ sudo nvidia-smi -mig 0 +``` + +Enable the virtual functions for the physical GPU in the `sysfs` file system. + +```sh +$ sudo /usr/lib/nvidia/sriov-manage -e 0000:41:00.0 +``` + +Get the `BDF` of the available virtual function on the GPU, and choose one for the +following steps. + +```sh +$ cd /sys/bus/pci/devices/0000:41:00.0/ +$ ls -l | grep virtfn +``` + +#### List all available vGPU instances + +The following shell snippet will walk the `sysfs` and only print instances +that are available, that can be created. + +```sh +# The 00.0 is often the PF of the device the VFs will have the funciont in the +# BDF incremented by some values so e.g. the very first VF is 0000:41:00.4 + +cd /sys/bus/pci/devices/0000:41:00.0/ + +for vf in $(ls -d virtfn*) +do + BDF=$(basename $(readlink -f $vf)) + for md in $(ls -d $vf/mdev_supported_types/*) + do + AVAIL=$(cat $md/available_instances) + NAME=$(cat $md/name) + DIR=$(basename $md) + + if [ $AVAIL -gt 0 ]; then + echo "| BDF | INSTANCES | NAME | DIR |" + echo "+--------------+-----------+----------------+------------+" + printf "| %12s |%10d |%15s | %10s |\n\n" "$BDF" "$AVAIL" "$NAME" "$DIR" + fi + + done +done +``` + +If there are available instances you get something like this (for the first VF), +beware that the output is highly dependent on the GPU you have, if there is no +output check again if `MIG` is really disabled. + +```sh +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-4C | nvidia-692 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-8C | nvidia-693 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-10C | nvidia-694 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-16C | nvidia-695 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-20C | nvidia-696 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-40C | nvidia-697 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 | GRID A100D-80C | nvidia-698 | + +``` + +Change to the `mdev_supported_types` directory for the virtual function on which +you want to create the `vGPU`. Taking the first output as an example: + +```sh +$ cd virtfn0/mdev_supported_types/nvidia-692 +$ UUIDGEN=$(uuidgen) +$ sudo bash -c "echo $UUIDGEN > create" +``` + +Confirm that the `vGPU` was created. You should see the `UUID` pointing to a +subdirectory of the `sysfs` space. + +```sh +$ ls -l /sys/bus/mdev/devices/ +``` + +Get the `IOMMU` group number and verify there is a `VFIO` device created to use +with Kata. + +```sh +$ ls -l /sys/bus/mdev/devices/*/ +$ ls -l /dev/vfio +``` + +Use the `VFIO` device created in the same way as in the pass-through use-case. +Beware that the guest needs the NVIDIA guest drivers, so one would need to build +a new guest `OS` image. + +### NVIDIA vGPU MIG-backed + +We're not going into detail what `MIG` is but briefly it is a technology to +partition the hardware into independent instances with guaranteed quality of +service. For more details see [NVIDIA Multi-Instance GPU User Guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/). + +First enable `MIG` mode for a GPU, depending on the platform you're running +a reboot would be necessary. Some platforms support GPU reset. + +```sh +$ sudo nvidia-smi -mig 1 +``` + +If the platform supports a GPU reset one can run, otherwise you will get a +warning to reboot the server. + +```sh +$ sudo nvidia-smi --gpu-reset +``` + +The driver per default provides a number of profiles that users can opt-in when +configuring the MIG feature. + +```sh +$ sudo nvidia-smi mig -lgip ++-----------------------------------------------------------------------------+ +| GPU instance profiles: | +| GPU Name ID Instances Memory P2P SM DEC ENC | +| Free/Total GiB CE JPEG OFA | +|=============================================================================| +| 0 MIG 1g.10gb 19 7/7 9.50 No 14 0 0 | +| 1 0 0 | ++-----------------------------------------------------------------------------+ +| 0 MIG 1g.10gb+me 20 1/1 9.50 No 14 1 0 | +| 1 1 1 | ++-----------------------------------------------------------------------------+ +| 0 MIG 2g.20gb 14 3/3 19.50 No 28 1 0 | +| 2 0 0 | ++-----------------------------------------------------------------------------+ + ... +``` + +Create the GPU instances that correspond to the `vGPU` types of the `MIG-backed` +`vGPUs` that you will create [NVIDIA A100 PCIe 80GB Virtual GPU Types](https://docs.nvidia.com/grid/13.0/grid-vgpu-user-guide/index.html#vgpu-types-nvidia-a100-pcie-80gb). + +```sh +# MIG 1g.10gb --> vGPU A100D-1-10C +$ sudo nvidia-smi mig -cgi 19 +``` + +List the GPU instances and get the GPU instance id to create the compute +instance. + +```sh +$ sudo nvidia-smi mig -lgi # list the created GPU instances +$ sudo nvidia-smi mig -cci -gi 9 # each GPU instance can have several compute + # instances. Instance -> Workload +``` + +Verify that the compute instances were created within the GPU instance + +```sh +$ nvidia-smi + ... snip ... ++-----------------------------------------------------------------------------+ +| MIG devices: | ++------------------+----------------------+-----------+-----------------------+ +| GPU GI CI MIG | Memory-Usage | Vol| Shared | +| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG| +| | | ECC| | +|==================+======================+===========+=======================| +| 0 9 0 0 | 0MiB / 9728MiB | 14 0 | 1 0 0 0 0 | +| | 0MiB / 4095MiB | | | ++------------------+----------------------+-----------+-----------------------+ + ... snip ... +``` + +We can use the [snippet](#list-all-available-vgpu-instances) from before to list +the available `vGPU` instances, this time `MIG-backed`. + +```sh +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.4 | 1 |GRID A100D-1-10C | nvidia-699 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:00.5 | 1 |GRID A100D-1-10C | nvidia-699 | + +| BDF | INSTANCES | NAME | DIR | ++--------------+-----------+----------------+------------+ +| 0000:41:01.6 | 1 |GRID A100D-1-10C | nvidia-699 | + ... snip ... +``` + +Repeat the steps after the [snippet](#list-all-available-vgpu-instances) listing +to create the corresponding `mdev` device and use the guest `OS` created in the +previous section with `time-sliced` `vGPUs`. ## Install NVIDIA Driver + Toolkit in Kata Containers Guest OS @@ -263,7 +482,7 @@ export EXTRA_PKGS="gcc make curl gnupg" ``` Having the `$ROOTFS_DIR` exported in the previous step we can now install all the -need parts in the guest OS. In this case we have an Ubuntu based rootfs. +needed parts in the guest OS. In this case, we have an Ubuntu based rootfs. First off all mount the special filesystems into the rootfs @@ -281,9 +500,9 @@ Now we can enter `chroot` $ sudo chroot ${ROOTFS_DIR} ``` -Inside the rootfs one is going to install the drivers and toolkit to enable easy -creation of GPU containers with Kata. We can also use this rootfs for any other -container not specifically only for GPUs. +Inside the rootfs one is going to install the drivers and toolkit to enable the +easy creation of GPU containers with Kata. We can also use this rootfs for any +other container not specifically only for GPUs. As a prerequisite install the copied kernel development packages @@ -304,6 +523,7 @@ $ ./NVIDIA-Linux-x86_64-510.54.run -x $ cd NVIDIA-Linux-x86_64-510.54 $ ./nvidia-installer -k 5.15.23-nvidia-gpu ``` + Having the drivers installed we need to install the toolkit which will take care of providing the right bits into the container. @@ -325,7 +545,7 @@ Create the hook execution file for Kata: /usr/bin/nvidia-container-toolkit -debug $@ ``` -As a last step one can do some cleanup of files or package caches. Build the +As the last step one can do some cleanup of files or package caches. Build the rootfs and configure it for use with Kata according to the development guide. Enable the `guest_hook_path` in Kata's `configuration.toml` @@ -334,7 +554,7 @@ Enable the `guest_hook_path` in Kata's `configuration.toml` guest_hook_path = "/usr/share/oci/hooks" ``` -One has build a NVIDIA rootfs, kernel and now we can run any GPU container +One has built a NVIDIA rootfs, kernel and now we can run any GPU container without installing the drivers into the container. Check NVIDIA device status with `nvidia-smi` @@ -362,7 +582,7 @@ Fri Mar 18 10:36:59 2022 +-----------------------------------------------------------------------------+ ``` -As a last step one can remove the additional packages and files that were added +As the last step one can remove the additional packages and files that were added to the `$ROOTFS_DIR` to keep it as small as possible. ## References From 8b57bf97ab726e1c05cb31be90765adf4ee5ae93 Mon Sep 17 00:00:00 2001 From: Snir Sheriber Date: Tue, 31 May 2022 15:28:05 +0300 Subject: [PATCH 26/35] workflows: add workflow_dispatch triggering to test-kata-deploy This will allow to trigger the test-kata-deploy workflow manually from any branch instead of using always the one that is defined on main See: https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/ Fixes: #4349 Signed-off-by: Snir Sheriber --- .github/workflows/kata-deploy-test.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/kata-deploy-test.yaml b/.github/workflows/kata-deploy-test.yaml index 4a9c06de1..e0d6afd7c 100644 --- a/.github/workflows/kata-deploy-test.yaml +++ b/.github/workflows/kata-deploy-test.yaml @@ -1,4 +1,5 @@ on: + workflow_dispatch: # this is used to trigger the workflow on non-main branches issue_comment: types: [created, edited] From 2e04833fb947c3358b68d99b43e4db5aa49bc41b Mon Sep 17 00:00:00 2001 From: Gabriela Cervantes Date: Wed, 1 Jun 2022 14:41:00 +0000 Subject: [PATCH 27/35] docs: Update Intel QAT documentation links This PR updates some Intel QAT documentation url links. Fixes #4374 Signed-off-by: Gabriela Cervantes --- docs/use-cases/using-Intel-QAT-and-kata.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/use-cases/using-Intel-QAT-and-kata.md b/docs/use-cases/using-Intel-QAT-and-kata.md index 3a7e6cf68..0bcd2e3bb 100644 --- a/docs/use-cases/using-Intel-QAT-and-kata.md +++ b/docs/use-cases/using-Intel-QAT-and-kata.md @@ -312,7 +312,7 @@ working properly with the Kata Containers VM. ### Build OpenSSL Intel® QAT engine container -Use the OpenSSL Intel® QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/master/demo/openssl-qat-engine) +Use the OpenSSL Intel® QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/main/demo/openssl-qat-engine) to build a container image with an optimized OpenSSL engine for Intel® QAT. Using `docker build` with the Kata Containers runtime can sometimes have issues. Therefore, make sure that `runc` is the default Docker container @@ -444,7 +444,7 @@ $ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest $ sudo ctr -n=k8s.io images import openssl-qat-engine.tar ``` -The [Intel® QAT Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/cmd/qat_plugin/README.md) +The [Intel® QAT Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/qat_plugin/README.md) needs to be started so that the virtual functions can be discovered and used by Kubernetes. From 002f2cd1098bc0983c4fdc7098ff1aadb1001969 Mon Sep 17 00:00:00 2001 From: "James O. D. Hunt" Date: Thu, 26 May 2022 15:55:26 +0100 Subject: [PATCH 28/35] snap: Use helper script and cleanup Move the common shell code to a helper script that is sourced by all parts. Add extra quoting to some variables in the snap config file and simplify. Fixes: #4304. Signed-off-by: James O. D. Hunt --- snap/local/snap-common.sh | 114 +++++++++++++++++++++++ snap/snapcraft.yaml | 189 +++++++++++++++----------------------- 2 files changed, 187 insertions(+), 116 deletions(-) create mode 100644 snap/local/snap-common.sh diff --git a/snap/local/snap-common.sh b/snap/local/snap-common.sh new file mode 100644 index 000000000..0a2a18e15 --- /dev/null +++ b/snap/local/snap-common.sh @@ -0,0 +1,114 @@ +#!/usr/bin/env bash +# +# Copyright (c) 2022 Intel Corporation +# +# SPDX-License-Identifier: Apache-2.0 + +# Description: Idempotent script to be sourced by all parts in a +# snapcraft config file. + +set -o errexit +set -o nounset +set -o pipefail + +# XXX: Bash-specific code. zsh doesn't support this option and that *does* +# matter if this script is run sourced... since it'll be using zsh! ;) +[ -n "$BASH_VERSION" ] && set -o errtrace + +[ -n "${DEBUG:-}" ] && set -o xtrace + +die() +{ + echo >&2 "ERROR: $0: $*" +} + +[ -n "${SNAPCRAFT_STAGE:-}" ] ||\ + die "must be sourced from a snapcraft config file" + +snap_yq_version=3.4.1 + +snap_common_install_yq() +{ + export yq="${SNAPCRAFT_STAGE}/bin/yq" + + local yq_pkg + yq_pkg="github.com/mikefarah/yq" + + local yq_url + yq_url="https://${yq_pkg}/releases/download/${snap_yq_version}/yq_${goos}_${goarch}" + curl -o "${yq}" -L "${yq_url}" + chmod +x "${yq}" +} + +# Function that should be called for each snap "part" in +# snapcraft.yaml. +snap_common_main() +{ + # Architecture + arch="$(uname -m)" + + case "${arch}" in + aarch64) + goarch="arm64" + qemu_arch="${arch}" + ;; + + ppc64le) + goarch="ppc64le" + qemu_arch="ppc64" + ;; + + s390x) + goarch="${arch}" + qemu_arch="${arch}" + ;; + + x86_64) + goarch="amd64" + qemu_arch="${arch}" + ;; + + *) die "unsupported architecture: ${arch}" ;; + esac + + dpkg_arch=$(dpkg --print-architecture) + + # golang + # + # We need the O/S name in golang format, but since we don't + # know if the godeps part has run, we don't know if golang is + # available yet, hence fall back to a standard system command. + goos="$(go env GOOS &>/dev/null || true)" + [ -z "$goos" ] && goos=$(uname -s|tr '[A-Z]' '[a-z]') + + export GOROOT="${SNAPCRAFT_STAGE}" + export GOPATH="${GOROOT}/gopath" + export GO111MODULE="auto" + + mkdir -p "${GOPATH}/bin" + export PATH="${GOPATH}/bin:${PATH}" + + # Proxy + export http_proxy="${http_proxy:-}" + export https_proxy="${https_proxy:-}" + + # Binaries + mkdir -p "${SNAPCRAFT_STAGE}/bin" + + export PATH="$PATH:${SNAPCRAFT_STAGE}/bin" + + # YAML query tool + export yq="${SNAPCRAFT_STAGE}/bin/yq" + + # Kata paths + export kata_dir=$(printf "%s/src/github.com/%s/%s" \ + "${GOPATH}" \ + "${SNAPCRAFT_PROJECT_NAME}" \ + "${SNAPCRAFT_PROJECT_NAME}") + + export versions_file="${kata_dir}/versions.yaml" + + [ -n "${yq:-}" ] && [ -x "${yq:-}" ] || snap_common_install_yq +} + +snap_common_main diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml index fd0101db5..badfc99ee 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -19,6 +19,8 @@ parts: - git - git-extras override-pull: | + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + version="9999" if echo "${GITHUB_REF:-}" | grep -q -E "^refs/tags"; then @@ -29,9 +31,6 @@ parts: snapcraftctl set-grade "stable" snapcraftctl set-version "${version}" - # setup GOPATH - this repo dir should be there - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} mkdir -p $(dirname ${kata_dir}) ln -sf $(realpath "${SNAPCRAFT_STAGE}/..") ${kata_dir} @@ -43,28 +42,12 @@ parts: build-packages: - curl override-build: | + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + # put everything in stage - cd ${SNAPCRAFT_STAGE} + cd "${SNAPCRAFT_STAGE}" - mkdir -p "${SNAPCRAFT_STAGE}/bin/" - yq_path="${SNAPCRAFT_STAGE}/bin/yq" - yq_pkg="github.com/mikefarah/yq" - goos="linux" - case "$(uname -m)" in - aarch64) goarch="arm64";; - ppc64le) goarch="ppc64le";; - x86_64) goarch="amd64";; - s390x) goarch="s390x";; - *) echo "unsupported architecture: $(uname -m)"; exit 1;; - esac - - yq_version=3.4.1 - yq_url="https://${yq_pkg}/releases/download/${yq_version}/yq_${goos}_${goarch}" - curl -o "${yq_path}" -L "${yq_url}" - chmod +x "${yq_path}" - - kata_dir=gopath/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - version="$(${yq_path} r ${kata_dir}/versions.yaml languages.golang.meta.newest-version)" + version="$(${yq} r ${kata_dir}/versions.yaml languages.golang.meta.newest-version)" tarfile="go${version}.${goos}-${goarch}.tar.gz" curl -LO https://golang.org/dl/${tarfile} tar -xf ${tarfile} --strip-components=1 @@ -81,28 +64,17 @@ parts: - uidmap - gnupg2 override-build: | - [ "$(uname -m)" = "ppc64le" ] || [ "$(uname -m)" = "s390x" ] && sudo apt-get --no-install-recommends install -y protobuf-compiler + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" - yq=${SNAPCRAFT_STAGE}/bin/yq + [ "${arch}" = "ppc64le" ] || [ "${arch}" = "s390x" ] && sudo apt-get --no-install-recommends install -y protobuf-compiler - # set GOPATH - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - - export GOROOT=${SNAPCRAFT_STAGE} - export PATH="${GOROOT}/bin:${PATH}" - export GO111MODULE="auto" - - http_proxy=${http_proxy:-""} - https_proxy=${https_proxy:-""} if [ -n "$http_proxy" ]; then echo "Setting proxy $http_proxy" - sudo -E systemctl set-environment http_proxy=$http_proxy || true - sudo -E systemctl set-environment https_proxy=$https_proxy || true + sudo -E systemctl set-environment http_proxy="$http_proxy" || true + sudo -E systemctl set-environment https_proxy="$https_proxy" || true fi # Copy yq binary. It's used in the container - mkdir -p "${GOPATH}/bin/" cp -a "${yq}" "${GOPATH}/bin/" echo "Unmasking docker service" @@ -113,63 +85,54 @@ parts: echo "Starting docker" sudo -E systemctl start docker || true - cd ${kata_dir}/tools/osbuilder + cd "${kata_dir}/tools/osbuilder" # build image export AGENT_INIT=yes export USE_DOCKER=1 export DEBUG=1 - arch="$(uname -m)" initrd_distro=$(${yq} r -X ${kata_dir}/versions.yaml assets.initrd.architecture.${arch}.name) image_distro=$(${yq} r -X ${kata_dir}/versions.yaml assets.image.architecture.${arch}.name) case "$arch" in x86_64) # In some build systems it's impossible to build a rootfs image, try with the initrd image - sudo -E PATH=$PATH make image DISTRO=${image_distro} || sudo -E PATH=$PATH make initrd DISTRO=${initrd_distro} + sudo -E PATH=$PATH make image DISTRO="${image_distro}" || sudo -E PATH="$PATH" make initrd DISTRO="${initrd_distro}" ;; aarch64|ppc64le|s390x) - sudo -E PATH=$PATH make initrd DISTRO=${initrd_distro} + sudo -E PATH="$PATH" make initrd DISTRO="${initrd_distro}" ;; - *) echo "unsupported architecture: $(uname -m)"; exit 1;; + *) die "unsupported architecture: ${arch}" ;; esac # Install image - kata_image_dir=${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers - mkdir -p ${kata_image_dir} - cp kata-containers*.img ${kata_image_dir} + kata_image_dir="${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers" + mkdir -p "${kata_image_dir}" + cp kata-containers*.img "${kata_image_dir}" runtime: after: [godeps, image, cloud-hypervisor] plugin: nil build-attributes: [no-patchelf] override-build: | - # set GOPATH - export GOPATH=${SNAPCRAFT_STAGE}/gopath - export GOROOT=${SNAPCRAFT_STAGE} - export PATH="${GOROOT}/bin:${PATH}" - export GO111MODULE="auto" - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" - cd ${kata_dir}/src/runtime + cd "${kata_dir}/src/runtime" - # setup arch - arch=$(uname -m) - if [ ${arch} = "ppc64le" ]; then - arch="ppc64" - fi + qemu_cmd="qemu-system-${qemu_arch}" # build and install runtime make \ - PREFIX=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr \ + PREFIX="/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr" \ SKIP_GO_VERSION_CHECK=1 \ - QEMUCMD=qemu-system-$arch + QEMUCMD="${qemu_cmd}" + make install \ PREFIX=/usr \ - DESTDIR=${SNAPCRAFT_PART_INSTALL} \ + DESTDIR="${SNAPCRAFT_PART_INSTALL}" \ SKIP_GO_VERSION_CHECK=1 \ - QEMUCMD=qemu-system-$arch + QEMUCMD="${qemu_cmd}" if [ ! -f ${SNAPCRAFT_PART_INSTALL}/../../image/install/usr/share/kata-containers/kata-containers.img ]; then sed -i -e "s|^image =.*|initrd = \"/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr/share/kata-containers/kata-containers-initrd.img\"|" \ @@ -186,44 +149,37 @@ parts: - bison - flex override-build: | - yq=${SNAPCRAFT_STAGE}/bin/yq - export PATH="${PATH}:${SNAPCRAFT_STAGE}" - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - versions_file="${kata_dir}/versions.yaml" + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + kernel_version="$(${yq} r $versions_file assets.kernel.version)" #Remove extra 'v' - kernel_version=${kernel_version#v} + kernel_version="${kernel_version#v}" - [ "$(uname -m)" = "s390x" ] && sudo apt-get --no-install-recommends install -y libssl-dev + [ "${arch}" = "s390x" ] && sudo apt-get --no-install-recommends install -y libssl-dev - export GOPATH=${SNAPCRAFT_STAGE}/gopath - export GO111MODULE="auto" - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - - cd ${kata_dir}/tools/packaging/kernel + cd "${kata_dir}/tools/packaging/kernel" kernel_dir_prefix="kata-linux-" # Setup and build kernel - ./build-kernel.sh -v ${kernel_version} -d setup + ./build-kernel.sh -v "${kernel_version}" -d setup cd ${kernel_dir_prefix}* make -j $(($(nproc)-1)) EXTRAVERSION=".container" - kernel_suffix=${kernel_version}.container - kata_kernel_dir=${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers - mkdir -p ${kata_kernel_dir} + kernel_suffix="${kernel_version}.container" + kata_kernel_dir="${SNAPCRAFT_PART_INSTALL}/usr/share/kata-containers" + mkdir -p "${kata_kernel_dir}" # Install bz kernel - make install INSTALL_PATH=${kata_kernel_dir} EXTRAVERSION=".container" || true - vmlinuz_name=vmlinuz-${kernel_suffix} - ln -sf ${vmlinuz_name} ${kata_kernel_dir}/vmlinuz.container + make install INSTALL_PATH="${kata_kernel_dir}" EXTRAVERSION=".container" || true + vmlinuz_name="vmlinuz-${kernel_suffix}" + ln -sf "${vmlinuz_name}" "${kata_kernel_dir}/vmlinuz.container" # Install raw kernel - vmlinux_path=vmlinux - [ "$(uname -m)" = "s390x" ] && vmlinux_path=arch/s390/boot/compressed/vmlinux - vmlinux_name=vmlinux-${kernel_suffix} - cp ${vmlinux_path} ${kata_kernel_dir}/${vmlinux_name} - ln -sf ${vmlinux_name} ${kata_kernel_dir}/vmlinux.container + vmlinux_path="vmlinux" + [ "${arch}" = "s390x" ] && vmlinux_path="arch/s390/boot/compressed/vmlinux" + vmlinux_name="vmlinux-${kernel_suffix}" + cp "${vmlinux_path}" "${kata_kernel_dir}/${vmlinux_name}" + ln -sf "${vmlinux_name}" "${kata_kernel_dir}/vmlinux.container" qemu: plugin: make @@ -250,12 +206,8 @@ parts: - libselinux1-dev - ninja-build override-build: | - yq=${SNAPCRAFT_STAGE}/bin/yq - export GOPATH=${SNAPCRAFT_STAGE}/gopath - export GO111MODULE="auto" - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" - versions_file="${kata_dir}/versions.yaml" branch="$(${yq} r ${versions_file} assets.hypervisor.qemu.version)" url="$(${yq} r ${versions_file} assets.hypervisor.qemu.url)" commit="" @@ -263,11 +215,11 @@ parts: patches_version_dir="${kata_dir}/tools/packaging/qemu/patches/tag_patches/${branch}" # download source - qemu_dir=${SNAPCRAFT_STAGE}/qemu + qemu_dir="${SNAPCRAFT_STAGE}/qemu" rm -rf "${qemu_dir}" git clone --depth 1 --branch ${branch} --single-branch ${url} "${qemu_dir}" - cd ${qemu_dir} - [ -z "${commit}" ] || git checkout ${commit} + cd "${qemu_dir}" + [ -z "${commit}" ] || git checkout "${commit}" [ -n "$(ls -A ui/keycodemapdb)" ] || git clone --depth 1 https://github.com/qemu/keycodemapdb ui/keycodemapdb/ [ -n "$(ls -A capstone)" ] || git clone --depth 1 https://github.com/qemu/capstone capstone @@ -278,10 +230,10 @@ parts: ${kata_dir}/tools/packaging/scripts/apply_patches.sh "${patches_version_dir}" # Only x86_64 supports libpmem - [ "$(uname -m)" = "x86_64" ] && sudo apt-get --no-install-recommends install -y apt-utils ca-certificates libpmem-dev + [ "${arch}" = "x86_64" ] && sudo apt-get --no-install-recommends install -y apt-utils ca-certificates libpmem-dev - configure_hypervisor=${kata_dir}/tools/packaging/scripts/configure-hypervisor.sh - chmod +x ${configure_hypervisor} + configure_hypervisor="${kata_dir}/tools/packaging/scripts/configure-hypervisor.sh" + chmod +x "${configure_hypervisor}" # static build. The --prefix, --libdir, --libexecdir, --datadir arguments are # based on PREFIX and set by configure-hypervisor.sh echo "$(PREFIX=/snap/${SNAPCRAFT_PROJECT_NAME}/current/usr ${configure_hypervisor} -s kata-qemu) \ @@ -291,17 +243,17 @@ parts: # Copy QEMU configurations (Kconfigs) case "${branch}" in "v5.1.0") - cp -a ${kata_dir}/tools/packaging/qemu/default-configs/* default-configs + cp -a "${kata_dir}"/tools/packaging/qemu/default-configs/* default-configs ;; *) - cp -a ${kata_dir}/tools/packaging/qemu/default-configs/* configs/devices/ + cp -a "${kata_dir}"/tools/packaging/qemu/default-configs/* configs/devices/ ;; esac # build and install make -j $(($(nproc)-1)) - make install DESTDIR=${SNAPCRAFT_PART_INSTALL} + make install DESTDIR="${SNAPCRAFT_PART_INSTALL}" prime: - -snap/ - -usr/bin/qemu-ga @@ -321,11 +273,13 @@ parts: plugin: nil after: [godeps] override-build: | + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + # Currently, only one platform uses the new rust virtiofsd. The # others make use of QEMU's C implementation. # # See "tools/packaging/scripts/configure-hypervisor.sh". - if [ "$(uname -m)" = 'x86_64' ] + if [ "${arch}" = 'x86_64' ] then echo "INFO: Building rust version of virtiofsd" else @@ -334,14 +288,8 @@ parts: exit 0 fi - # put everything in stage - cd ${SNAPCRAFT_STAGE} - - export PATH="$PATH:${SNAPCRAFT_STAGE}/bin" - export GOPATH=${SNAPCRAFT_STAGE}/gopath - - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} cd "${kata_dir}" + # Download the rust implementation of virtiofsd tools/packaging/static-build/virtiofsd/build-static-virtiofsd.sh sudo install \ @@ -356,22 +304,31 @@ parts: plugin: nil after: [godeps] override-build: | - arch=$(uname -m) + source "${SNAPCRAFT_PROJECT_DIR}/snap/local/snap-common.sh" + if [ "${arch}" == "aarch64" ] || [ "${arch}" == "x86_64" ]; then sudo apt-get -y update sudo apt-get -y install ca-certificates curl gnupg lsb-release - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg - echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + curl -fsSL https://download.docker.com/linux/ubuntu/gpg |\ + sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg + distro_codename=$(lsb_release -cs) + echo "deb [arch=${dpkg_arch} signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu ${distro_codename} stable" |\ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get -y update sudo apt-get -y install docker-ce docker-ce-cli containerd.io sudo systemctl start docker.socket - export GOPATH=${SNAPCRAFT_STAGE}/gopath - kata_dir=${GOPATH}/src/github.com/${SNAPCRAFT_PROJECT_NAME}/${SNAPCRAFT_PROJECT_NAME} - cd ${kata_dir} + cd "${SNAPCRAFT_PROJECT_DIR}" sudo -E NO_TTY=true make cloud-hypervisor-tarball - tar xvJpf build/kata-static-cloud-hypervisor.tar.xz -C /tmp/ - install -D /tmp/opt/kata/bin/cloud-hypervisor ${SNAPCRAFT_PART_INSTALL}/usr/bin/cloud-hypervisor + + tarfile="${SNAPCRAFT_PROJECT_DIR}/tools/packaging/kata-deploy/local-build/build/kata-static-cloud-hypervisor.tar.xz" + tmpdir=$(mktemp -d) + + tar -xvJpf "${tarfile}" -C "${tmpdir}" + + install -D "${tmpdir}/opt/kata/bin/cloud-hypervisor" "${SNAPCRAFT_PART_INSTALL}/usr/bin/cloud-hypervisor" + + rm -rf "${tmpdir}" fi apps: From c1476a174bee746508d36bbb2fbddea99d4941b7 Mon Sep 17 00:00:00 2001 From: Snir Sheriber Date: Tue, 31 May 2022 15:44:12 +0300 Subject: [PATCH 29/35] docs: update release process with latest workflow triggering instructions Signed-off-by: Snir Sheriber --- docs/Release-Process.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Release-Process.md b/docs/Release-Process.md index 9dfc37041..7e8dfe329 100644 --- a/docs/Release-Process.md +++ b/docs/Release-Process.md @@ -48,7 +48,7 @@ ### Merge all bump version Pull requests - The above step will create a GitHub pull request in the Kata projects. Trigger the CI using `/test` command on each bump Pull request. - - Trigger the test-kata-deploy workflow on the kata-containers repository bump Pull request using `/test_kata_deploy` (monitor under the "action" tab). + - Trigger the `test-kata-deploy` workflow which is under the `Actions` tab on the repository GitHub page (make sure to select the correct branch and validate it passes). - Check any failures and fix if needed. - Work with the Kata approvers to verify that the CI works and the pull requests are merged. From b9fc24ff3a8fd4b38bd9f10ff4d299b7fe959131 Mon Sep 17 00:00:00 2001 From: Snir Sheriber Date: Tue, 31 May 2022 15:46:17 +0300 Subject: [PATCH 30/35] docs: update release process github token instructions and fix the gpg generating key url Signed-off-by: Snir Sheriber --- docs/Release-Process.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Release-Process.md b/docs/Release-Process.md index 7e8dfe329..7dcfb84a3 100644 --- a/docs/Release-Process.md +++ b/docs/Release-Process.md @@ -4,11 +4,11 @@ ## Requirements - [hub](https://github.com/github/hub) - * Using an [application token](https://github.com/settings/tokens) is required for hub. + * Using an [application token](https://github.com/settings/tokens) is required for hub (set to a GITHUB_TOKEN environment variable). - GitHub permissions to push tags and create releases in Kata repositories. -- GPG configured to sign git tags. https://help.github.com/articles/generating-a-new-gpg-key/ +- GPG configured to sign git tags. https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key - You should configure your GitHub to use your ssh keys (to push to branches). See https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/. * As an alternative, configure hub to push and fork with HTTPS, `git config --global hub.protocol https` (Not tested yet) * From 38a3188206d1949a0934064af81379cef3bd2187 Mon Sep 17 00:00:00 2001 From: Chen Yiyang Date: Thu, 2 Jun 2022 11:22:20 +0800 Subject: [PATCH 31/35] runk: Support `list` sub-command Support list sub-command. It will traverse the root directory, parse status file and print basic information of containers. Behavior and print format consistent with runc. To handle race with runk delete or system user modify, the loop will continue to traverse when errors are encountered. Fixes: #4362 Signed-off-by: Chen Yiyang --- src/tools/runk/Cargo.lock | 27 +++++++++++ src/tools/runk/Cargo.toml | 2 + src/tools/runk/src/commands/list.rs | 69 +++++++++++++++++++++++++++++ src/tools/runk/src/commands/mod.rs | 1 + src/tools/runk/src/main.rs | 1 + 5 files changed, 100 insertions(+) create mode 100644 src/tools/runk/src/commands/list.rs diff --git a/src/tools/runk/Cargo.lock b/src/tools/runk/Cargo.lock index f8628cbf7..a1691a796 100644 --- a/src/tools/runk/Cargo.lock +++ b/src/tools/runk/Cargo.lock @@ -985,7 +985,9 @@ dependencies = [ "serde_json", "slog", "slog-async", + "tabwriter", "tokio", + "users", ] [[package]] @@ -1166,6 +1168,15 @@ dependencies = [ "unicode-xid", ] +[[package]] +name = "tabwriter" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "36205cfc997faadcc4b0b87aaef3fbedafe20d38d4959a7ca6ff803564051111" +dependencies = [ + "unicode-width", +] + [[package]] name = "take_mut" version = "0.2.2" @@ -1348,12 +1359,28 @@ version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7e8820f5d777f6224dc4be3632222971ac30164d4a258d595640799554ebfd99" +[[package]] +name = "unicode-width" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ed742d4ea2bd1176e236172c8429aaf54486e7ac098db29ffe6529e0ce50973" + [[package]] name = "unicode-xid" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3" +[[package]] +name = "users" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24cc0f6d6f267b73e5a2cadf007ba8f9bc39c6a6f9666f8cf25ea809a153b032" +dependencies = [ + "libc", + "log", +] + [[package]] name = "version_check" version = "0.9.4" diff --git a/src/tools/runk/Cargo.toml b/src/tools/runk/Cargo.toml index 83aca82ca..973ae21c7 100644 --- a/src/tools/runk/Cargo.toml +++ b/src/tools/runk/Cargo.toml @@ -22,6 +22,8 @@ slog-async = "2.7.0" tokio = { version = "1.15.0", features = ["full"] } serde = { version = "1.0.133", features = ["derive"] } serde_json = "1.0.74" +users = "0.11.0" +tabwriter = "1.2.1" [workspace] members = [ diff --git a/src/tools/runk/src/commands/list.rs b/src/tools/runk/src/commands/list.rs new file mode 100644 index 000000000..dbd427755 --- /dev/null +++ b/src/tools/runk/src/commands/list.rs @@ -0,0 +1,69 @@ +// Copyright 2021-2022 Kata Contributors +// +// SPDX-License-Identifier: Apache-2.0 +// + +use super::state::get_container_state_name; +use anyhow::Result; +use libcontainer::status::{get_current_container_state, Status}; +use liboci_cli::List; +use oci::ContainerState; +use slog::{info, Logger}; +use std::{fs, os::unix::prelude::MetadataExt, path::Path}; +use std::{io, io::Write}; +use tabwriter::TabWriter; +use users::get_user_by_uid; + +pub fn run(_: List, root: &Path, logger: &Logger) -> Result<()> { + let mut content = String::new(); + for entry in fs::read_dir(root)? { + let entry = entry?; + // Possibly race with runk delete, so continue loop when any error occurs below + let metadata = match entry.metadata() { + Ok(metadata) => metadata, + Err(_) => continue, + }; + if !metadata.is_dir() { + continue; + } + let container_id = match entry.file_name().into_string() { + Ok(id) => id, + Err(_) => continue, + }; + let status = match Status::load(root, &container_id) { + Ok(status) => status, + Err(_) => continue, + }; + let state = match get_current_container_state(&status) { + Ok(state) => state, + Err(_) => continue, + }; + // Just like runc, pid of stopped container is 0 + let pid = match state { + ContainerState::Stopped => 0, + _ => status.pid, + }; + // May replace get_user_by_uid with getpwuid(3) + let owner = match get_user_by_uid(metadata.uid()) { + Some(user) => String::from(user.name().to_string_lossy()), + None => format!("#{}", metadata.uid()), + }; + content.push_str(&format!( + "{}\t{}\t{}\t{}\t{}\t{}\n", + container_id, + pid, + get_container_state_name(state), + status.bundle.display(), + status.created, + owner + )); + } + + let mut tab_writer = TabWriter::new(io::stdout()); + writeln!(&mut tab_writer, "ID\tPID\tSTATUS\tBUNDLE\tCREATED\tOWNER")?; + write!(&mut tab_writer, "{}", content)?; + tab_writer.flush()?; + + info!(&logger, "list command finished successfully"); + Ok(()) +} diff --git a/src/tools/runk/src/commands/mod.rs b/src/tools/runk/src/commands/mod.rs index 12017263d..216686506 100644 --- a/src/tools/runk/src/commands/mod.rs +++ b/src/tools/runk/src/commands/mod.rs @@ -6,6 +6,7 @@ pub mod create; pub mod delete; pub mod kill; +pub mod list; pub mod run; pub mod spec; pub mod start; diff --git a/src/tools/runk/src/main.rs b/src/tools/runk/src/main.rs index 8f464f438..0171a229f 100644 --- a/src/tools/runk/src/main.rs +++ b/src/tools/runk/src/main.rs @@ -78,6 +78,7 @@ async fn cmd_run(subcmd: SubCommand, root_path: &Path, logger: &Logger) -> Resul SubCommand::Common(cmd) => match cmd { CommonCmd::Run(run) => commands::run::run(run, root_path, logger).await, CommonCmd::Spec(spec) => commands::spec::run(spec, logger), + CommonCmd::List(list) => commands::list::run(list, root_path, logger), _ => { return Err(anyhow!("command is not implemented yet")); } From 168f325c43b520e599a5a02c1c4205bb49cc7fd1 Mon Sep 17 00:00:00 2001 From: Gabriela Cervantes Date: Mon, 30 May 2022 15:12:50 +0000 Subject: [PATCH 32/35] docs: Update configuration reference for snap documentation This PR updates the url link for the kata containers configuration for the general snap documentation. Fixes #4341 Signed-off-by: Gabriela Cervantes --- snap/local/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/snap/local/README.md b/snap/local/README.md index 47174d174..4b449ef17 100644 --- a/snap/local/README.md +++ b/snap/local/README.md @@ -92,7 +92,7 @@ and [configured][7]. [1]: https://docs.snapcraft.io/snaps/intro [2]: ../../docs/design/architecture/README.md#root-filesystem-image [3]: https://docs.snapcraft.io/reference/confinement#classic -[4]: https://github.com/kata-containers/runtime#configuration +[4]: https://github.com/kata-containers/kata-containers/tree/main/src/runtime#configuration [5]: https://docs.docker.com/engine/reference/commandline/dockerd [6]: ../../docs/install/docker/ubuntu-docker-install.md [7]: ../../docs/Developer-Guide.md#configure-to-use-initrd-or-rootfs-image From 9726f56fdc8e6b3beb55d21d121ca08c941acb10 Mon Sep 17 00:00:00 2001 From: Feng Wang Date: Wed, 1 Jun 2022 13:14:20 -0700 Subject: [PATCH 33/35] runtime: force stop container after the container process exits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Set thestop container force flag to true so that the container state is always set to “StateStopped” after the container wait goroutine is finished. This is necessary for the following delete container step to succeed. Fixes: #4359 Signed-off-by: Feng Wang --- src/runtime/pkg/containerd-shim-v2/wait.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/runtime/pkg/containerd-shim-v2/wait.go b/src/runtime/pkg/containerd-shim-v2/wait.go index 0f6c4fe06..5e86149c1 100644 --- a/src/runtime/pkg/containerd-shim-v2/wait.go +++ b/src/runtime/pkg/containerd-shim-v2/wait.go @@ -78,7 +78,7 @@ func wait(ctx context.Context, s *service, c *container, execID string) (int32, shimLog.WithField("sandbox", s.sandbox.ID()).Error("failed to delete sandbox") } } else { - if _, err = s.sandbox.StopContainer(ctx, c.id, false); err != nil { + if _, err = s.sandbox.StopContainer(ctx, c.id, true); err != nil { shimLog.WithError(err).WithField("container", c.id).Warn("stop container failed") } } From 9d27c1fced805981a6327a539006a266291d4b31 Mon Sep 17 00:00:00 2001 From: Feng Wang Date: Wed, 1 Jun 2022 17:38:28 -0700 Subject: [PATCH 34/35] agent: ignore ESRCH error when destroying containers destroy() method should ignore the ESRCH error from signal::kill and continue the operation as ESRCH is often considered harmless. Fixes: #4359 Signed-off-by: Feng Wang --- src/agent/rustjail/src/container.rs | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/src/agent/rustjail/src/container.rs b/src/agent/rustjail/src/container.rs index 88c058868..a1207104f 100644 --- a/src/agent/rustjail/src/container.rs +++ b/src/agent/rustjail/src/container.rs @@ -1063,7 +1063,19 @@ impl BaseContainer for LinuxContainer { let st = self.oci_state()?; for pid in self.processes.keys() { - signal::kill(Pid::from_raw(*pid), Some(Signal::SIGKILL))?; + match signal::kill(Pid::from_raw(*pid), Some(Signal::SIGKILL)) { + Err(Errno::ESRCH) => { + info!( + self.logger, + "kill encounters ESRCH, pid: {}, container: {}", + pid, + self.id.clone() + ); + continue; + } + Err(err) => return Err(anyhow!(err)), + Ok(_) => continue, + } } if spec.hooks.is_some() { From fe3c1d9cdd3bacf545deca09c626a0907e6283a9 Mon Sep 17 00:00:00 2001 From: Gabriela Cervantes Date: Mon, 6 Jun 2022 14:48:34 +0000 Subject: [PATCH 35/35] docs: Update storage documentation link This PR updates the storage documentation link for the devicemapper snapshotter. Fixes #4398 Signed-off-by: Gabriela Cervantes --- docs/design/architecture/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/architecture/storage.md b/docs/design/architecture/storage.md index 974f260c3..8cdda1e2e 100644 --- a/docs/design/architecture/storage.md +++ b/docs/design/architecture/storage.md @@ -20,7 +20,7 @@ For virtio-fs, the [runtime](README.md#runtime) starts one `virtiofsd` daemon ## Devicemapper The -[devicemapper `snapshotter`](https://github.com/containerd/containerd/tree/master/snapshots/devmapper) +[devicemapper `snapshotter`](https://github.com/containerd/containerd/tree/main/snapshots/devmapper) is a special case. The `snapshotter` uses dedicated block devices rather than formatted filesystems, and operates at the block level rather than the file level. This knowledge is used to directly use the