Commit Graph

3398 Commits

Author SHA1 Message Date
Greg Kurz
4443bb68a4 Merge pull request #4064 from tiezhuoyu/4063/no-need-to-write-error-of-virtiofsd-to-kata-log
runtime: no need to write virtiofsd error to log
2022-04-13 11:59:19 +02:00
Hyounggyu Choi
d136c9c240 test: Fix golangci-lint error for s390x
This is to fix a test failure for the
kata-containers-2.0-ubuntu-20.04-s390x-main-baseline jenkins job

Fixes: #4088

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
2022-04-13 09:20:51 +02:00
Fupan Li
66aa07649b Merge pull request #4062 from liubin/fix/4061-add-links-for-kata-monitor
kata-monitor: add some links when generating pages for browsers
2022-04-13 11:30:21 +08:00
Francesco Giudici
86977ff780 kata-monitor: update the hrefs in the debug/pprof index page
kata-monitor allows to get data profiles from the kata shim
instances running on the same node by acting as a proxy
(e.g., http://$NODE_ADDRESS:8090/debug/pprof/?sandbox=$MYSANDBOXID).
In order to proxy the requests and the responses to the right shim,
kata-monitor requires to pass the sandbox id via a query string in the
url.

The profiling index page proxied by kata-monitor contains the link to all
the data profiles available. All the links anyway do not contain the
sandbox id included in the request: the links result then broken when
accessed through kata-monitor.
This happens because the profiling index page comes from the kata shim,
which will not include the query string provided in the http request.

Let's add on-the-fly the sandbox id in each href tag returned by the kata
shim index page before providing the proxied page.

Fixes: #4054

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2022-04-12 15:53:59 +02:00
Fabiano Fidêncio
78f30c33c6 agent: Avoid agent panic when reading empty stats
This was seen in an issue report, where we'd try to unwrap a None value,
leading to a panic.

Fixes: #4077
Related: #4043

Full backtrace:
```
"thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', rustjail/src/cgroups/fs/mod.rs:593:31"
"stack backtrace:"
"   0:     0x7f0390edcc3a - std::backtrace_rs::backtrace::libunwind::trace::hd5eff4de16dbdd15"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5"
"   1:     0x7f0390edcc3a - std::backtrace_rs::backtrace::trace_unsynchronized::h04a775b4c6ab90d6"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5"
"   2:     0x7f0390edcc3a - std::sys_common::backtrace::_print_fmt::h3253c3db9f17d826"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:67:5"
"   3:     0x7f0390edcc3a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h02bfc712fc868664"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:46:22"
"   4:     0x7f0390a91fbc - core::fmt::write::hfd5090d1132106d8"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/fmt/mod.rs:1149:17"
"   5:     0x7f0390edb804 - std::io::Write::write_fmt::h34acb699c6d6f5a9"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/io/mod.rs:1697:15"
"   6:     0x7f0390edbee0 - std::sys_common::backtrace::_print::hfca761479e3d91ed"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:49:5"
"   7:     0x7f0390edbee0 - std::sys_common::backtrace::print::hf666af0b87d2b5ba"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:36:9"
"   8:     0x7f0390edbee0 - std::panicking::default_hook::{{closure}}::hb4617bd1d4a09097"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:211:50"
"   9:     0x7f0390edb2da - std::panicking::default_hook::h84f684d9eff1eede"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:228:9"
"  10:     0x7f0390edb2da - std::panicking::rust_panic_with_hook::h8e784f5c39f46346"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:606:17"
"  11:     0x7f0390f0c416 - std::panicking::begin_panic_handler::{{closure}}::hef496869aa926670"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:500:13"
"  12:     0x7f0390f0c3b6 - std::sys_common::backtrace::__rust_end_short_backtrace::h8e9b039b8ed3e70f"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:139:18"
"  13:     0x7f0390f0c372 - rust_begin_unwind"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5"
"  14:     0x7f03909062c0 - core::panicking::panic_fmt::h568976b83a33ae59"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14"
"  15:     0x7f039090641c - core::panicking::panic::he2e71cfa6548cc2c"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:48:5"
"  16:     0x7f0390eb443f - <rustjail::cgroups::fs::Manager as rustjail::cgroups::Manager>::get_stats::h85031fc1c59c53d9"
"  17:     0x7f03909c0138 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hfa6e6cd7516f8d11"
"  18:     0x7f0390d697e5 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hffbaa534cfa97d44"
"  19:     0x7f039099c0b3 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hae3ab083a06d0b4b"
"  20:     0x7f0390af9e1e - std::panic::catch_unwind::h1fdd25c8ebba32e1"
"  21:     0x7f0390b7c4e6 - tokio::runtime::task::raw::poll::hd3ebbd0717dac808"
"  22:     0x7f0390f49f3f - tokio::runtime::thread_pool::worker::Context::run_task::hfdd63cd1e0b17abf"
"  23:     0x7f0390f3a599 - tokio::runtime::task::raw::poll::h62954f6369b1d210"
"  24:     0x7f0390f37863 - std::sys_common::backtrace::__rust_begin_short_backtrace::h1c58f232c078bfe9"
"  25:     0x7f0390f4f3dd - core::ops::function::FnOnce::call_once{{vtable.shim}}::h2d329a84c0feed57"
"  26:     0x7f0390f0e535 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h137e5243c6233a3b"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/boxed.rs:1694:9"
"  27:     0x7f0390f0e535 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h7331c46863d912b7"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/boxed.rs:1694:9"
"  28:     0x7f0390f0e535 - std::sys::unix:🧵:Thread:🆕:thread_start::h1fb20b966cb927ab"
"                               at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys/unix/thread.rs:106:17"
```

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
2022-04-12 11:19:08 +02:00
Zhuoyu Tie
6e79042aa0 runtime: no need to write virtiofsd error to log
The scanner reads nothing from viriofsd stderr pipe, because param
'--syslog' rediercts stderr to syslog. So there is no need to write
scanner.Text() to kata log

Fixes: #4063

Signed-off-by: Zhuoyu Tie <tiezhuoyu@outlook.com>
2022-04-12 15:59:57 +08:00
Braden Rayhorn
9b6f24b2ee agent: add tests for mount_to_rootfs function
Add test coverage for mount_to_rootfs function in src/mount.rs.
Includes minor refactoring to make function more easily testable.

Fixes #4073

Signed-off-by: Braden Rayhorn <bradenrayhorn@fastmail.com>
2022-04-11 21:42:38 -05:00
Braden Rayhorn
c3776b1792 agent: add tests for is_signal_handled function
Add test coverage for is_signal_handled function in rpc.rs. Includes
refactors to make the function testable and handle additional cases.

Fixes #3939

Signed-off-by: Braden Rayhorn <bradenrayhorn@fastmail.com>
2022-04-11 21:23:55 -05:00
Braden Rayhorn
9c22d9554e agent: add tests for update_container_namespaces
Add test coverage for update_container_namespaces function
in src/rpc.rs. Includes minor refactor to make function easier
to test.

Fixes #4034

Signed-off-by: Braden Rayhorn <bradenrayhorn@fastmail.com>
2022-04-11 18:27:30 -05:00
Yibo Zhuang
92c00c7e84 agent: fsGroup support for direct-assigned volume
Adding two functions set_ownership and
recursive_ownership_change to support changing group id
ownership for a mounted volume.

The set_ownership will be called in common_storage_handler
after mount_storage performs the mount for the volume.
set_ownership will be a noop if the FSGroup field in the
Storage struct is not set which indicates no chown will be
performed. If FSGroup field is specified, then it will
perform the recursive walk of the mounted volume path to
change ownership of all files and directories to the
desired group id. It will also configure the SetGid bit
so that files created the directory will have group
following parent directory group.

If the fsGroupChangePolicy is on root mismatch,
then the group ownership will be skipped if the root
directory group id alreasy matches the desired group
id and if the SetGid bit is also set on the root directory.

This is the same behavior as what
Kubelet does today when performing the recursive walk
to change ownership.

Fixes #4018

Signed-off-by: Yibo Zhuang <yibzhuang@gmail.com>
2022-04-11 08:57:13 -07:00
Yibo Zhuang
532d53977e runtime: fsGroup support for direct-assigned volume
The fsGroup will be specified by the fsGroup key in
the direct-assign mountinfo metadate field.
This will be set when invoking the kata-runtime
binary and providing the key, value pair in the metadata
field. Similarly, the fsGroupChangePolicy will also
be provided in the mountinfo metadate field.

Adding an extra fields FsGroup and FSGroupChangePolicy
in the Mount construct for container mount which will
be populated when creating block devices by parsing
out the mountInfo.json.

And in handleDeviceBlockVolume of the kata-agent client,
it checks if the mount FSGroup is not nil, which
indicates that fsGroup change is required in the guest,
and will provide the FSGroup field in the protobuf to
pass the value to the agent.

Fixes #4018

Signed-off-by: Yibo Zhuang <yibzhuang@gmail.com>
2022-04-11 08:41:13 -07:00
Yibo Zhuang
6a47b82c81 proto: fsGroup support for direct-assigned volume
This change adds two fields to the Storage pb

FSGroup which is a group id that the runtime
specifies to indicate to the agent to perform a
chown of the mounted volume to the specified
group id after mounting is complete in the guest.

FSGroupChangePolicy which is a policy to indicate
whether to always perform the group id ownership
change or only if the root directory group id
does not match with the desired group id.

These two fields will allow CSI plugins to indicate
to Kata that after the block device is mounted in
the guest, group id ownership change should be performed
on that volume.

Fixes #4018

Signed-off-by: Yibo Zhuang <yibzhuang@gmail.com>
2022-04-11 08:41:13 -07:00
Braden Rayhorn
9d5e7ee0d4 agent: add tests for mount_storage
Add test coverage for mount_storage function in src/mount.rs.

Fixes: #4068

Signed-off-by: Braden Rayhorn <bradenrayhorn@fastmail.com>
2022-04-10 21:42:20 -05:00
bin
f8cc5d1ad8 kata-monitor: add some links when generating pages for browsers
Add some links to rendered webpages for better user experience,
let users can jump to pages only by clicking links in browsers.

Fixes: #4061

Signed-off-by: bin <bin@hyper.sh>
2022-04-11 09:29:56 +08:00
Fabiano Fidêncio
698e45f403 Merge pull request #4057 from bradenrayhorn/test-parse-mount-flags-and-options
agent: add test coverage for parse_mount_flags_and_options function
2022-04-08 14:42:18 +02:00
Fabiano Fidêncio
761e8313de Merge pull request #3985 from bradenrayhorn/test-do-write-stream
agent: add tests for do_write_stream function
2022-04-08 14:34:57 +02:00
Peng Tao
4f551e3428 Merge pull request #4048 from liubin/fix/3303-delete-virtiofsd-debug-option
runtime: delete debug option in virtiofsd
2022-04-08 15:42:38 +08:00
Peng Tao
a83a16e32c Merge pull request #4059 from garrettmahin/test-process-grpc-to-oci
rustjail: add test coverage for process_grpc_to_oci function
2022-04-08 15:39:28 +08:00
Peng Tao
95e45fab38 Merge pull request #4053 from ManaSugi/fix-makefile-for-features
agent: Allow the agent to be rebuilt with the change of Cargo features
2022-04-08 15:38:25 +08:00
garrettmahin
c31cd0e81a rustjail: add test coverage for process_grpc_to_oci function
Add test coverage for the process_grpc_to_oci function in src/rustjail/lib.rs

Fixes #4058

Signed-off-by: Garrett Mahin <garrett.mahin@gmail.com>
2022-04-07 20:50:48 -05:00
Bin Liu
9c1c219a3f Merge pull request #4007 from liubin/fix/3959-add-csi-rs-to-gitignore
protocols: add src/csi.rs to .gitignore
2022-04-08 09:33:04 +08:00
Braden Rayhorn
1118a3d2da agent: add test coverage for parse_mount_flags_and_options function
Add test coverage for the parse_mount_flags_and_options function
in src/mount.rs.

Fixes #4056

Signed-off-by: Braden Rayhorn <bradenrayhorn@fastmail.com>
2022-04-07 17:46:35 -05:00
bin
9d5b03a1b7 runtime: delete debug option in virtiofsd
virtiofsd's debug will be enabled if hypervisor's debug has been
enabled, this will generate too many noisy logs from virtiofsd.

Unbind the relationship of log level between virtiofsd and
hypervisor, if users want to see debug log of virtiofsd,
can set it by:

  virtio_fs_extra_args = ["-o", "log_level=debug"]

Fixes: #3303

Signed-off-by: bin <bin@hyper.sh>
2022-04-07 19:55:22 +08:00
Manabu Sugimoto
eff7c7e0ff agent: Allow the agent to be rebuilt with the change of Cargo features
This allows the kata-agent to be rebuilt when Cargo "features" is
changed. The Makefile for the agent do not need to specify the
sources for prerequisites by having Cargo check for the sources
changes.

Fixes: #4052

Signed-off-by: Manabu Sugimoto <Manabu.Sugimoto@sony.com>
2022-04-07 20:09:20 +09:00
Greg Kurz
d0d3787233 Merge pull request #3696 from shippomx/main
kata-runtime enable hugepage support
2022-04-06 16:47:04 +02:00
Jaylyn Ren
b975f2e8d2 Virtcontainers: Enable hot plugging vhost-user-blk device on ARM
The vhost-user-blk can be hotplugged on the PCI bridge successfully on
X86, but failed on Arm. However, hotplugging it on Root Port as a PCIe
device can work well on ARM.
Open the "pcie_root_port" in configuration.toml is needed.

Fixes: #4019

Signed-off-by: Jaylyn Ren <jaylyn.ren@arm.com>
2022-04-06 17:37:51 +08:00
bin
962d05ec86 protocols: add src/csi.rs to .gitignore
After running make in src/agent, the git working area will be changed:

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	src/libs/protocols/src/csi.rs

The generated file by `build.rs` should be ignored in git.

Fixes: #3959

Signed-off-by: bin <bin@hyper.sh>
2022-04-06 09:55:38 +08:00
Fabiano Fidêncio
b39caf43f1 Merge pull request #3923 from Jakob-Naucke/no-initrd-se
runtime: Allow and require no initrd for SE
2022-04-05 09:26:07 +02:00
Feng Wang
354cd3b9b6 runtime: Base64 encode the direct volume mountInfo path
This is to avoid accidentally deleting multiple volumes.

Fixes #4020

Signed-off-by: Feng Wang <feng.wang@databricks.com>
2022-04-04 19:56:46 -07:00
Braden Rayhorn
485aeabb6b agent: add tests for do_write_stream function
Add test coverage for do_write_stream function of AgentService
in src/rpc.rs. Includes minor refactoring to make function more
easily testable.

Fixes #3984

Signed-off-by: Braden Rayhorn <bradenrayhorn@fastmail.com>
2022-04-04 08:21:01 -05:00
Archana Shinde
e62bc8e7f3 Merge pull request #3915 from Juneezee/test/t.TempDir
test: use `T.TempDir` to create temporary test directory
2022-04-04 01:34:46 -07:00
Fabiano Fidêncio
8980d04e25 Merge pull request #4023 from fidencio/wip/expose-service-offload-option-to-clh
clh: Expose service offload configuration
2022-04-01 14:10:33 +02:00
Fabiano Fidêncio
98750d792b clh: Expose service offload configuration
This configuration option is valid for all the hypervisor that are going
to be used with the confidential containers effort, thus exposing the
configuration option for Cloud Hypervisor as well.

Fixes: #4022

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
2022-04-01 11:15:55 +02:00
Bin Liu
416cc90b7a Merge pull request #3972 from wfly1998/main
agent: use ms as unit of cputime instead of ticks
2022-04-01 15:34:06 +08:00
Bin Liu
5d0adb2164 Merge pull request #3995 from wxx213/main
agent: fix container stop error with signal SIGRTMIN+3
2022-04-01 11:29:14 +08:00
Wang Xingxing
0d765bd082 agent: fix container stop error with signal SIGRTMIN+3
The nix::sys::signal::Signal package api cannot deal with SIGRTMIN+3,
directly use libc function to send the signal.

Fixes: #3990

Signed-off-by: Wang Xingxing <stellarwxx@163.com>
2022-03-31 10:49:45 +08:00
Eng Zer Jun
59c7165ee1 test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

This commit also updates the unit test advice to use `T.TempDir` to
create temporary directory in tests.

Fixes: #3924

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-31 09:31:36 +08:00
snir911
18dc578134 Merge pull request #3999 from fgiudici/kata-monitor_fix_help
kata-monitor: fix duplicated output when printing usage
2022-03-30 18:56:59 +03:00
Francesco Giudici
a63bbf9793 kata-monitor: fix duplicated output when printing usage
(default: "/run/containerd/containerd.sock") is duplicated when
printing kata-monitor usage:

[root@kubernetes ~]# kata-monitor --help
Usage of kata-monitor:
  -listen-address string
        The address to listen on for HTTP requests. (default ":8090")
  -log-level string
        Log level of logrus(trace/debug/info/warn/error/fatal/panic). (default "info")
  -runtime-endpoint string
        Endpoint of CRI container runtime service. (default: "/run/containerd/containerd.sock") (default "/run/containerd/containerd.sock")

the golang flag package takes care of adding the defaults when printing
usage. Remove the explicit print of the value so that it would not be
printed on screen twice.

Fixes: #3998

Signed-off-by: Francesco Giudici <fgiudici@redhat.com>
2022-03-30 11:58:53 +02:00
bin
5e1c30d484 runtime: add logs around sandbox monitor
For debugging purposes, add some logs.

Fixes: #3815

Signed-off-by: bin <bin@hyper.sh>
2022-03-29 16:59:12 +08:00
bin
fb8be96194 runtime: stop getting OOM events when ttrpc: closed error
getOOMEvents is a long-waiting call, it will retry when failed.
For cases of agent shutdown, the retry should stop.

When the agent hasn't detected agent has died, we can also check
whether the error is "ttrpc: closed".

Fixes: #3815

Signed-off-by: bin <bin@hyper.sh>
2022-03-29 16:39:01 +08:00
Bin Liu
9495316145 Merge pull request #3962 from yaoyinnan/fix/3750-VirtioMem
runtime: Remove the explicit VirtioMem set and fix the comment
2022-03-29 10:20:05 +08:00
yaoyinnan
66f05c5bcb runtime: Remove the explicit VirtioMem set and fix the comment
Modify the 2Mib in the comment to 4Mib.
VirtioMem is set by configuration file or annotation. And setupVirtioMem is called only when VirtioMem is true.

Fixes: #3750

Signed-off-by: yaoyinnan <yaoyinnan@foxmail.com>
2022-03-28 21:21:38 +08:00
Yu Li
800e4a9cfb agent: use ms as unit of cputime instead of ticks
For the library `procfs`, the unit of values in `CpuTime` is ticks,
and we do not know how many ticks per second from metrics because the
`tps` in `CpuTime` is private.

But there are some implements in `CpuTime` for getting these values,
e.g., `user_ms()` for `user`, and `nice_ms()` for `nice`.  With these
values, accurate time can be obtained.

Fixes: #3979

Acked-by: zhaojizhuang <571130360@qq.com>
Signed-off-by: Yu Li <liyu.yukiteru@bytedance.com>
2022-03-28 19:30:09 +08:00
Feng Wang
0928eb9f4e agent: Kill the all the container processes of the same cgroup
Otherwise the container process might leak and cause an unclean exit

Fixes: #3913

Signed-off-by: Feng Wang <feng.wang@databricks.com>
2022-03-27 10:06:58 -07:00
Jakob Naucke
ff17c756d2 runtime: Allow and require no initrd for SE
Previously, it was not permitted to have neither an initrd nor an image.
However, this is the exact config to use for Secure Execution, where the
initrd is part of the image to be specified as `-kernel`. Require the
configuration of no initrd for Secure Execution.

Also
- remove redundant code for image/initrd checking -- no need to check in
  `newQemuHypervisorConfig` (calling) when it is also checked in
  `getInitrdAndImage` (called)
- use `QemuCCWVirtio` constant when possible

Fixes: #3922
Signed-off-by: Jakob Naucke <jakob.naucke@ibm.com>
2022-03-25 18:36:12 +01:00
Feng Wang
19f372b5f5 runtime: Add more debug logs for container io stream copy
This can help debugging container lifecycle issues

Fixes: #3913

Signed-off-by: Feng Wang <feng.wang@databricks.com>
2022-03-24 21:35:16 -07:00
Peng Tao
098374b179 Merge pull request #3934 from dcmiddle/fix-agent-check
Agent: fix unneeded late initialization lint
2022-03-24 16:02:11 +08:00
David Gibson
c77e34de33 runtime: Move mock hook source
src/runtime/virtcontainers/hook/mock contains a simple example hook in Go.
The only thing this is used for is for some tests in
src/runtime/pkg/katautils/hook_test.go.  It doesn't really have anything
to do with the rest of the virtcontainers package.

So, move it next to the test code that uses it.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-03-23 19:37:35 +11:00
David Gibson
86723b51ae virtcontainers: Remove unused install/uninstall targets
We've now removed the need to install the mock hook binary for unit tests.
However, it turns out that managing that was the *only* thing that the
install and uninstall targets in the virtcontainers Makefile handled.

So, remove them.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-03-23 19:37:18 +11:00