From f4efe6c899803caf1f0bcf1a24ad01b59bbe13c9 Mon Sep 17 00:00:00 2001 From: Christian Decker Date: Tue, 20 Dec 2022 13:05:02 +0100 Subject: [PATCH] docs: Add docs on code generation --- doc/dev/contributors/codegen.md | 138 ++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 doc/dev/contributors/codegen.md diff --git a/doc/dev/contributors/codegen.md b/doc/dev/contributors/codegen.md new file mode 100644 index 000000000..3c05d4fd7 --- /dev/null +++ b/doc/dev/contributors/codegen.md @@ -0,0 +1,138 @@ +# Code Generation + +The CLN project has a multitude of interfaces, most of which are +generated from an abstract schema: + + - Wire format for peer-to-peer communication: this is the binary + format that is specific by the [LN spec][spec]. It uses the + [generate-wire.py][generate-wire.py] script to parse the (faux) CSV + files that are automatically extrated from the specification and + writes C source code files that are then used internally to encode + and decode messages, as well as provide print functions for the + messages. + + - Wire format for inter-daemon communication: CLN follows a + multi-daemon architecture, making communication explicit across + daemons. For this inter-daemon communication we use a slightly + altered message format from the [LN spec][spec]. The changes are 1) + addition of FD passing semantics to allow establishing a new + connection between daemons (communication uses + [socketpair][socketpair]s, so no `connect`), and 2) change the + message length prefix from `u16` to `u32`, allowing for messages + larger than 65Kb. The CSV files are with the respective sub-daemon + and also use [generate-wire.py][generate-wire.py] to generate + encoding, decoding and printing functions. + + - We describe the JSON-RPC using [JSON Schema][jschema] in the + [`doc/schemas`][doc-schemas] directory. Each method has a + `.request.json` for the request message, and a `.schema.json` for + the response (the mismatch is historical and will eventually be + addressed). During tests the `pytest` target will verify responses, + however the JSON-RPC methods are _not_ generated (yet?). We do + generate various client stubs for languages, using the + [`msggen`][msggen] tool. More on the generated stubs and utilities + below. + +## Man pages + +The [manpages][man] are partially generated from the JSON schemas +using the [`fromschema`][fromschema] tool. It reads the request schema +and fills in the manpage between two markers: + +```markdown +[comment]: # (GENERATE-FROM-SCHEMA-START) +... +[comment]: # (GENERATE-FROM-SCHEMA-END) +``` + +!!! note + + Some of this functionality overlaps with [`msggen`][msggen] (parsing the Schemas) + and [blockreplace.py][blockreplace.py] (filling in the template). It + is likely that this will eventually be merged. + +[blockreplace.py]: https://github.com/ElementsProject/lightning/blob/master/devtools/blockreplace.py +[man]: ../../reference/ +[fromschema]: https://github.com/ElementsProject/lightning/blob/master/tools/fromschema.py + +## `msggen` + +`msggen` is used to generate JSON-RPC client stubs, and converters +between in-memory formats and the JSON format. In addition, by +chaining some of these we can expose a [grpc][grpc] interface that +matches the JSON-RPC interface. This conversion chain is implemented +in the [grpc-plugin][grpc-plugin] + + +
+```mermaid +graph LR + A[JSON schema]; + A --> B[cln-rpc]; + B --> B1[Request Structs]; + B --> B2[Response Structs]; + B --> B3[Method stubs]; + + A --> C[cln-grpc]; + C --> C1[Protobuf File]; + C --> C2[In-memory conversion]; + C --> C3[Service Implementation]; +``` +
Artifacts generated from the JSON Schemas using `msggen`
+
+ +### `cln-rpc` + +We use `msggen` to generate the Rust bindings crate +[`cln-rpc`][cln-rpc]. These bindings contain the stubs for the +JSON-RPC methods, as well as types for the request and response +structs. The [generator code][cln-rpc-gen] maps each abstract JSON-RPC +type to a Rust type, minimizing size (e.g., binary data is +hex-decoded). + +The calling pattern follows the `call(req_obj) -> resp_obj` format, +and the individual arguments are not expanded. For more ergonomic +handling of generic requests and responses we also define the +`Request` and `Response` enumerations, so you can hand them to a +generic function without having to resort to dynamic dispatch. + +The remainder of the crate implements an async/await JSON-RPC client, +that can deal with the Unix Domain Socket [transport][man:json-rpc] +used by CLN. + +### `cln-grpc` + +The `cln-grpc` crate is mostly used to provide the primitives to build +the `grpc-plugin`. As mentioned above, the grpc functionality relies on a chain of generated parts: + + - First `msggen` is used to generate the [protobuf file][proto], + containing the service definition with the method stubs, and the types + referenced by those stubs. + - Next it generates the `convert.rs` file which is used to convert + the structs for in-memory representation from `cln-rpc` into the + corresponding protobuf structs. + - Finally `msggen` generates the `server.rs` file which can be bound + to a grpc endpoint listening for incoming grpc requests, and it + will convert the request and forward it to the JSON-RPC. Upon + receiving the response it gets converted back into a grpc response + and sent back. + +```mermaid +graph LR + A[grpc client] --> B[grpc server] -->|convert.rs| C[cln-rpc] --> D[lightningd]; + D --> C -->|convert.rs| B --> A; +``` + +[proto]: https://github.com/ElementsProject/lightning/blob/master/cln-grpc/proto/node.proto +[man:json-rpc]: ../../lightningd-rpc.7.md +[cln-rpc-gen]: https://github.com/ElementsProject/lightning/blob/master/contrib/msggen/msggen/gen/rust.py +[spec]: https://github.com/lightning/bolts +[generate-wire.py]: https://github.com/ElementsProject/lightning/blob/master/tools/generate-wire.py +[socketpair]: https://man7.org/linux/man-pages/man2/socketpair.2.html +[jschema]: https://json-schema.org/ +[doc-schemas]: https://github.com/ElementsProject/lightning/tree/master/doc/schemas +[msggen]: https://github.com/ElementsProject/lightning/tree/master/contrib/msggen +[grpc]: https://grpc.io/ +[cln-grpc]: https://docs.rs/cln-grpc/0.1.1/cln_grpc/ +[grpc-plugin]: https://github.com/ElementsProject/lightning/tree/master/plugins/grpc-plugin +[cln-rpc]: https://github.com/ElementsProject/lightning/tree/master/cln-rpc