In line with [other work](#127) for JSON support, this PR adds support
for [`json_array_length`](https://www.sqlite.org/json1.html#jarraylen).
This includes a first pass at supporting the JSON path for accessing
values within the JSON.
I've added tests in rust and tcl.
Closes#555
This PR's genesis is from investigating #532, but I still can't reliably
reproduce it on either `main` or this branch so I don't know if this PR
_fixes_ anything, but I guess it aligns us more with sqlite anyway
---
Anyway: I looked at DBs created with limbo and with sqlite using
[ImHex](https://github.com/WerWolv/ImHex) and the differences seem to
be:
1. SQLite uses varint according to [the
spec](https://www.sqlite.org/fileformat.html#record_format), whereas
limbo always encodes integers as i64
2. Limbo adds 4 bytes of zeros for overflow page pointer (even in cases
where the cell doesnt overflow)
3. Limbo adds a space after `CREATE TABLE name` before the `(` even when
user doesn't specify it?
I implemented the following:
- Fix 1: Varint serialization of i8, i16, i24, i32, i48 and i64
according to payload, instead of always using i64
- Fix 2: Removed the 4 bytes reserved for overflow page pointer in non-
overflow cases
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#550
small follow up to https://github.com/tursodatabase/limbo/pull/539
contains:
- Variable renaming and comments to `btreecursor.insert_into_cell()`
- New utility methods `pagecontent.header_size()`,
`pagecontent.cell_pointer_array_size()`,
`pagecontent.unallocated_region_start()` and
`pagecontent.unallocated_region_size()`
- Refactor of `btreecursor.compute_free_space()` (plus comments and
variable renaming)
- Rename `pagecontent.cell_get_raw_pointer_region()` to
`pagecontent.cell_pointer_array_offset_and_size()` and remove its usage
in `btreecursor.defragment_page()`
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#543
Closes#436
This PR fixes four issues:
1. Not respecting user-provided column names (e.g. `INSERT INTO foo
(b,c) values (1,2);` would just insert into the first two columns
regardless of what index `b` and `c` have)
2. Limbo would get in an infinite loop when inserting too many values
(too many i.e. more columns than the table has)
3. False positive unique constraint error on non-primary key columns
when inserting multiple values, e.g.
```
limbo> create table t1(v1 int);
limbo> insert into t1 values (1),(2);
Runtime error: UNIQUE constraint failed: t1.v1 (19)
```
as seen [here](https://github.com/tursodatabase/limbo/pull/490#issuecomm
ent-2545545562)
4. Limbo no longer uses a coroutine for INSERT when only inserting one
row. See [this comment](https://github.com/tursodatabase/limbo/issues/43
6#issuecomment-2533937845). For the equivalent query, Limbo now
generates:
```
limbo> EXPLAIN INSERT INTO users (name, email) VALUES ('John Doe', 'john@example.com');
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 10 0 0 Start at 10
1 OpenWriteAsync 0 2 0 0
2 OpenWriteAwait 0 0 0 0
3 String8 0 3 0 John Doe 0 r[3]='John Doe'
4 String8 0 4 0 john@example.com 0 r[4]='john@example.com'
5 NewRowId 0 1 0 0
6 MakeRecord 2 3 5 0 r[5]=mkrec(r[2..4])
7 InsertAsync 0 5 1 0
8 InsertAwait 0 0 0 0
9 Halt 0 0 0 0
10 Transaction 0 1 0 0
11 Null 0 2 0 0 r[2]=NULL
12 Goto 0 1 0 0
```
---
Note that this PR doesn't fix e.g. #472 which requires creating an index
on the non-rowid primary key column(s), nor does it implement rollback
(e.g. inserting two rows where one fails to unique constraint still
inserts the other row)
---
**EXAMPLES OF ERRONEOUS BEHAVIOR -- current head of main:**
wrong column inserted
```
limbo> create table rowidalias_b (a, b INTEGER PRIMARY KEY, c, d);
limbo> insert into rowidalias_b (d) values ('d only');
limbo> select * from rowidalias_b;
d only|1|| <-- gets inserted into column a
```
wrong column inserted
```
limbo> create table textpk (a, b text primary key, c);
limbo> insert into textpk (a,b,c) values ('a','b','c');
limbo> select * from textpk;
a|b|c
limbo> insert into textpk (b,c) values ('b','c');
limbo> select * from textpk;
a|b|c
b|c| <--- b gets inserted into column a
```
false positive from integer check due to attempting to insert wrong
column
```
limbo> create table rowidalias_b (a, b INTEGER PRIMARY KEY, c, d);
limbo> insert into rowidalias_b (a,c) values ('lol', 'bal');
Parse error: MustBeInt: the value in the register is not an integer <-- tries to insert c into b column
```
false positive from integer check due to attempting to insert wrong
column
```
limbo> CREATE TABLE users (
id INTEGER PRIMARY KEY,
name TEXT,
email TEXT
);
limbo> INSERT INTO users (name, email) VALUES ('John Doe', 'john@example.com');
Parse error: MustBeInt: the value in the register is not an integer. <-- tries to insert name into id column
```
allows write of nonexistent column
```
limbo> create table a(b);
limbo> insert into a (nonexistent_col) values (1);
limbo> select * from a;
1
```
hangs forever when inserting too many values
```
limbo> create table a (b integer primary key);
limbo> insert into a values (1,2); <-- spinloops forever at 100% cpu
```
unique constraint error on non-unique column
```
limbo> create table t1(v1 int);
limbo> insert into t1 values (1),(2);
Runtime error: UNIQUE constraint failed: t1.v1 (19)
```
**EXAMPLES OF CORRECT BEHAVIOR -- this branch:**
correct column inserted
```
limbo> create table rowidalias_b (a, b INTEGER PRIMARY KEY, c, d);
limbo> insert into rowidalias_b (d) values ('d only');
limbo> select * from rowidalias_b;
|1||d only
```
correct column inserted
```
limbo> create table textpk (a, b text primary key, c);
limbo> insert into textpk (a,b,c) values ('a','b','c');
limbo> select * from textpk;
a|b|c
limbo> insert into textpk (b,c) values ('b','c');
limbo> select * from textpk;
a|b|c
|b|c
```
correct columns inserted, PK autoincremented
```
limbo> create table rowidalias_b (a, b INTEGER PRIMARY KEY, c, d);
limbo> insert into rowidalias_b (a,c) values ('lol', 'bal');
limbo> select * from rowidalias_b;
lol|1|bal|
```
correct column inserted, PK autoincremented
```
limbo> CREATE TABLE users (
id INTEGER PRIMARY KEY,
name TEXT,
email TEXT
);
limbo> INSERT INTO users (name, email) VALUES ('John Doe', 'john@example.com');
limbo> select * from users;
1|John Doe|john@example.com
```
reports parse error correctly about wrong number of values
```
limbo> create table a (b integer primary key);
limbo> insert into a values (1,2);
Parse error: table a has 1 columns but 2 values were supplied
```
reports parse error correctly about nonexistent column
```
limbo> create table a(b);
limbo> insert into a (nonexistent_col) values (1);
Parse error: table a has no column named nonexistent_col
```
no unique constraint error on non-unique column
```
limbo> create table t1(v1 int);
limbo> insert into t1 values (1),(2);
limbo> select * from t1;
1
2
```
**Also, added multi-row inserts to simulator and ran into at least
this:**
```
Seed: 9444323279823516485
path to db '"/var/folders/qj/r6wpj6657x9cj_1jx_62cpgr0000gn/T/.tmpcYczRv/simulator.db"'
Initial opts SimulatorOpts { ticks: 3474, max_connections: 1, max_tables: 79, read_percent: 61, write_percent: 12, delete_percent: 27, max_interactions: 2940, page_size: 4096 }
thread 'main' panicked at core/storage/sqlite3_ondisk.rs:332:36:
called `Result::unwrap()` on an `Err` value: Corrupt("Invalid page type: 83")
```
Closes#533
This pr adds support for multiple readers and a single writer with a
custom made lock called `LimboRwLock`. Basically there are 5 allowed
read locks which store the max frame allowed in that "snapshot" and any
reader will try to acquire the biggest one possible. Writer will just
try to lock the `write_lock` and if not successful, it will return busy.
The only checkpoint mode supported for now is `PASSIVE` but it should be
trivial to add more modes.
This needs testing, but I will do it in another PR. I just wanted to do
it in another PR.
Closes#544
This PR makes two small incremental updates:
1- It adds a Clap CLI for simulator configuration, using the same Clap
version as the Limbo cli crate
2- It creates a new submodule called `simulator`, moving simulator
related structs from the large main file into their own files.
I am open to suggestions on the submodule name instead of `simulator` as
it's kind of weird to have `simulator/simulator` in the file tree.
Closes#540
This PR should have no functional changes, just variable renaming and
comments
Using `///` comment format for better IDE support
Reviewed-by: Pere Diaz Bou <penberg@iki.fi>
Closes#539
`cargo test` is always failing on FreeBSD, the following is one of the
errors:
```
---- tests::test_simple_overflow_page stdout ----
thread 'tests::test_simple_overflow_page' panicked at test/src/lib.rs:32:84:
called `Result::unwrap()` on an `Err` value: IOError(Os { code: 2, kind: NotFound, message: "No such file or directory" })
```
After some digging, I found that the `open_file` function in
`core/io/generic.rs` does not respect the `OpenFlags::Create` flag. This
commit adds support for file creation in the `open_file` function.
`cargo test` now passes on FreeBSD.
Closes#537
#509
Started the discussion on discord about possibly supporting UUID types
natively. This PR only implements the `sqlean` extension's functions and
behavior, with the only changes being:
1. uuid's are returned as `blob`s by default. (that was an assumption I
made considering perf, thinking this would be preferred if UUID ended up
being a supported native type. if `text` is preferred I can change it)
2. `uuidv7` types here can accept an argument of seconds since epoch to
customize the embedded timestamp. The func
`uuidv7_timestamp_ms(string_or_blob_v7)` allows the user to convert
their uuid7 back into the timestamp.
Closes#518
