The following code reproduces the leak, with memory usage increasing
over time:
```
#[tokio::main]
async fn main() {
let db = Builder::new_local(":memory:").build().await.unwrap();
let conn = db.connect().unwrap();
conn.execute("SELECT load_extension('./target/debug/liblimbo_series');", ())
.await
.unwrap();
loop {
conn.execute("SELECT * FROM generate_series(1,10,2);", ())
.await
.unwrap();
}
}
```
After switching to the system allocator, the leak becomes detectable
with Valgrind:
```
32,000 bytes in 1,000 blocks are definitely lost in loss record 24 of 24
at 0x538580F: malloc (vg_replace_malloc.c:446)
by 0x62E15FA: alloc::alloc::alloc (alloc.rs:99)
by 0x62E172C: alloc::alloc::Global::alloc_impl (alloc.rs:192)
by 0x62E1530: allocate (alloc.rs:254)
by 0x62E1530: alloc::alloc::exchange_malloc (alloc.rs:349)
by 0x62E0271: new<limbo_series::GenerateSeriesCursor> (boxed.rs:257)
by 0x62E0271: open_GenerateSeriesVTab (lib.rs:19)
by 0x425D8FA: limbo_core::VirtualTable::open (lib.rs:732)
by 0x4285DDA: limbo_core::vdbe::execute::op_vopen (execute.rs:890)
by 0x42351E8: limbo_core::vdbe::Program::step (mod.rs:396)
by 0x425C638: limbo_core::Statement::step (lib.rs:610)
by 0x40DB238: limbo::Statement::execute::{{closure}} (lib.rs:181)
by 0x40D9EAF: limbo::Connection::execute::{{closure}} (lib.rs:109)
by 0x40D54A1: example::main::{{closure}} (example.rs:26)
```
Interestingly, when using mimalloc, neither Valgrind nor mimalloc’s
internal statistics report the leak.
I keep having 3+ PR's in at the same time and always deal with crazy
conflicts because everything in the `ext` library is together in one
file.
This PR moves each category of extension into its own file, and
separates the `vfs` functionality in Core into the `ext/dynamic` module,
so that it can be more easily separated from wasm (or non feature =
"fs") targets to prevent build issues.
The only semantic changes made in this PR is the feature gating of vfs,
the rest is simply organizing and cleaning up imports.
Was unsure if `vfs` should be a feature on the `core` side too, or to
just enable it with the `fs` feature which seemed reasonable, as that
was already the current behavior. But let me know if we want it entirely
behind it's own feature.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#1124
This PR cleans up some comments in the extension API and prevents
extensions themselves from calling 'free' on Value types that are
exposed to the user facing traits, as well as changes the `from_ffi`
method for OwnedValues to take ownership and automatically free the
values to prevent memory leaks.
This PR also finds the name of the `args: &[Value]` argument for scalar
functions in extensions, and uses that in the proc macro, instead of
relying on documentation to communicate that the parameter must be named
`args`.
Closes#1054
This PR started out as one to improve the API of extensions but I ended
up building on top of this quite a bit and it just kept going. Sorry
this one is so large but there wasn't really a good stopping point, as
it kept leaving stuff in broken states.
**VCreate**: Support for `CREATE VIRTUAL TABLE t USING vtab_module`
**VUpdate**: Support for `INSERT` and `DELETE` methods on virtual
tables.
Sqlite uses `xUpdate` function with the `VUpdate` opcode to handle all
insert/update/delete functionality in virtual tables..
have to just document that:
```
if args[0] == NULL: INSERT args[1] the values in args[2..]
if args[1] == NULL: DELETE args[0]
if args[0] != NULL && len(args) > 2: Update values=args[2..] rowid=args[0]
```
I know I asked @jussisaurio on discord about this already, but it just
sucked so bad that I added some internal translation so we could expose
a [nice API](https://github.com/tursodatabase/limbo/pull/996/files#diff-
3e8f8a660b11786745b48b528222d11671e9f19fa00a032a4eefb5412e8200d1R54) and
handle the logic ourselves while keeping with sqlite's opcodes.
I'll change it back if I have to, I just thought it was genuinely awful
to have to rely on comments to explain all that to extension authors.
The included extension is not meant to be a legitimately useful one, it
is there for testing purposes. I did something similar in #960 using a
test extension, so I figure when they are both merged, I will go back
and combine them into one since you can do many kinds at once, and that
way it will reduce the amount of crates and therefore compile time.
1. Remaining opcodes.
2. `UPDATE` (when we support the syntax)
3. `xConnect` - expose API for a DB connection to a vtab so it can
perform arbitrary queries.
Closes#996
