Pekka Enberg
707d1a0705
Merge 'Update Flake and add Darwin link flags' from Ethan Niser
...
This pull request:
- updates the nix flake to include tcl 8.6 addressing #243
- updates the nix flake to add the darwin "core foundation" lib
- updates the make flake to link the core foundation lib on darwin
I was previously unable to run `make test-sqlite3` with this error:
https://gist.github.com/ethanniser/5b96888637da6ea9e9d56dcc3af9b573
But now it is able to run.
Closes #259
2024-08-02 08:12:14 +03:00
Ethan Niser
d47d494865
update flake to include tcl latest (addresses #243 ) and add core foundation lib and link flags for darwin
2024-08-01 18:00:46 -07:00
Pekka Enberg
e18fc511bb
scripts/merge-pr.py: Use temporal file for commit message
...
...fixes problems like eating perfectly valid Markdown and stuff from
pull request descriptions.
2024-08-01 21:07:39 +03:00
Pekka Enberg
64738d6348
Merge 'Add ability to annotate instructions with comments' from Jussi Saurio
...
Similar to what SQLite does.
```
limbo> EXPLAIN SELECT u.age, p.name FROM users u LEFT JOIN products p ON u.first_name = p.name;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 25 0 0 Start at 25
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 RewindAsync 0 0 0 0
6 RewindAwait 0 -5 0 0 Rewind table u
7 Integer 0 1 0 0 r[1]=0; init LEFT JOIN match flag
8 RewindAsync 1 0 0 0
9 RewindAwait 1 -11 0 0 Rewind table p
10 Column 0 1 2 0 r[2]=u.first_name
11 Column 1 1 3 0 r[3]=p.name
12 Ne 2 3 17 0 if r[2]!=r[3] goto 17
13 Integer 1 1 0 0 r[1]=1; record LEFT JOIN hit
14 Column 0 9 4 0 r[4]=u.age
15 Column 1 1 5 0 r[5]=p.name
16 ResultRow 4 2 0 0 output=r[4..5]
17 NextAsync 1 0 0 0
18 NextAwait 1 9 0 0
19 IfPos 1 22 0 0 r[1]>0 -> r[1]-=0, goto 22
20 NullRow 1 0 0 0 Set cursor 1 to a (pseudo) NULL row
21 Goto 0 13 0 0
22 NextAsync 0 0 0 0
23 NextAwait 0 6 0 0
24 Halt 0 0 0 0
25 Transaction 0 0 0 0
26 Goto 0 1 0 0
```
Closes #258
2024-08-01 21:07:31 +03:00
jussisaurio
20c085614f
Add ability to annotate instructions with comments
2024-08-01 20:34:45 +03:00
Pekka Enberg
fedb558346
Update COMPAT.md
2024-08-01 20:24:44 +03:00
Pekka Enberg
5d29e04854
Update COMPAT.md
2024-08-01 19:24:10 +03:00
Pekka Enberg
cb34deef09
Limbo 0.0.3
2024-08-01 18:57:13 +03:00
Pekka Enberg
6af9aafddc
Update CHANGELOG.md
2024-08-01 18:57:13 +03:00
Pekka Enberg
d97a434e29
Merge 'Use SeekRowId to avoid nested scans' from Jussi Saurio
...
The SQLite `SeekRowid` instruction tries to find a record in a B-tree table with a particular `rowid` (or INTEGER PRIMARY KEY, if defined), and jumps if it is not found. In this PR, constraints like `tbl.id = 5` or `tbl.id = tbl2.id` are transformed into special `SeekRowid` expressions that emit `SeekRowid` VM instructions. This avoids a table scan and instead completes in `log N` time (for a table of `N` rows), because of how B-Trees work.
`LoopInfo` now contains a `Plan` which is either `Scan` or `SeekRowid` -- in the case of `SeekRowid` no `Rewind`/`Next` instructions are emitted - i.e. no looping is done.
`BTreeCursor` now implements a `btree_seek_rowid()` method that tries to find a row by `rowid`.
Our loop order is currently static, i.e. `SELECT * from a join b join c` always results in "loop a, loop b, loop c", so `SeekRowId` is only supported for equality expressions where the non-PK side of the expression only refers to outer loops or constants. Examples:
**Because `u.id` refers to an outer loop compared to the primary key `p.id`, `p` is selected for SeekRowid optimization:**
```
limbo> explain SELECT u.age FROM users u JOIN products p ON u.id = p.id
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 14 0 0 Start at 14
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 RewindAsync 0 0 0 0
6 RewindAwait 0 -5 0 0 Rewind table u
7 RowId 0 1 0 0 r[1]=u.rowid
8 SeekRowid 1 1 11 0 if (r[1]!=p.rowid) goto 11
9 Column 0 9 2 0 r[2]=u.age
10 ResultRow 2 1 0 0 output=r[2]
11 NextAsync 0 0 0 0
12 NextAwait 0 6 0 0
13 Halt 0 0 0 0
14 Transaction 0 0 0 0
15 Goto 0 1 0 0
limbo> SELECT u.age FROM users u JOIN products p ON u.id = p.id
94
37
18
33
15
89
24
63
77
13
22
```
**Because `5` refers to a constant and `u.id` is a primary key, `u` is selected for SeekRowid optimization:**
```
limbo> explain SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = 5;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 15 0 0 Start at 15
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 5 1 0 0 r[1]=5
6 SeekRowid 0 1 14 0 if (r[1]!=u.rowid) goto 14
7 RewindAsync 1 0 0 0
8 RewindAwait 1 -8 0 0 Rewind table p
9 Column 0 1 2 0 r[2]=u.first_name
10 Column 1 1 3 0 r[3]=p.name
11 ResultRow 2 2 0 0 output=r[2..3]
12 NextAsync 1 0 0 0
13 NextAwait 1 8 0 0
14 Halt 0 0 0 0
15 Transaction 0 0 0 0
16 Goto 0 1 0 0
limbo> SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = 5;
Edward|hat
Edward|cap
Edward|shirt
Edward|sweater
Edward|sweatshirt
Edward|shorts
Edward|jeans
Edward|sneakers
Edward|boots
Edward|coat
Edward|accessories
```
**Same, but LEFT JOIN:**
```
limbo> EXPLAIN SELECT u.first_name, p.name FROM users u LEFT JOIN products p ON u.id = 5;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 20 0 0 Start at 20
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 0 1 0 0 r[1]=0
6 RewindAsync 1 0 0 0
7 RewindAwait 1 -11 0 0 Rewind table p
8 Integer 5 2 0 0 r[2]=5
9 SeekRowid 0 2 14 0 if (r[2]!=u.rowid) goto 14
10 Integer 1 1 0 0 r[1]=1
11 Column 0 1 3 0 r[3]=u.first_name
12 Column 1 1 4 0 r[4]=p.name
13 ResultRow 3 2 0 0 output=r[3..4]
14 NextAsync 1 0 0 0
15 NextAwait 1 7 0 0
16 IfPos 1 19 0 0 r[1]>0 -> r[1]-=0, goto 19
17 NullRow 1 0 0 0 Set cursor 1 to a (pseudo) NULL row
18 Goto 0 10 0 0
19 Halt 0 0 0 0
20 Transaction 0 0 0 0
21 Goto 0 1 0 0
limbo> SELECT u.first_name, p.name FROM users u LEFT JOIN products p ON u.id = 5;
Edward|hat
Edward|cap
Edward|shirt
Edward|sweater
Edward|sweatshirt
Edward|shorts
Edward|jeans
Edward|sneakers
Edward|boots
Edward|coat
Edward|accessories
```
**Both `p` and `u` selected for optimization:**
```
limbo> EXPLAIN SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = p.id and u.id = 5;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 13 0 0 Start at 13
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 5 1 0 0 r[1]=5
6 SeekRowid 0 1 12 0 if (r[1]!=u.rowid) goto 12
7 RowId 0 2 0 0 r[2]=u.rowid
8 SeekRowid 1 2 12 0 if (r[2]!=p.rowid) goto 12
9 Column 0 1 3 0 r[3]=u.first_name
10 Column 1 1 4 0 r[4]=p.name
11 ResultRow 3 2 0 0 output=r[3..4]
12 Halt 0 0 0 0
13 Transaction 0 0 0 0
14 Goto 0 1 0 0
limbo> SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = p.id and u.id = 5;
Edward|sweatshirt
```
**`p.id + 1` refers to an INNER loop compared to the primary key `u.id`, so optimization is skipped:**
```
limbo> EXPLAIN SELECT u.age FROM users u JOIN products p ON u.id = p.id + 1;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 21 0 0 Start at 21
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 RewindAsync 0 0 0 0
6 RewindAwait 0 -5 0 0 Rewind table u
7 RewindAsync 1 0 0 0
8 RewindAwait 1 -8 0 0 Rewind table p
9 RowId 0 1 0 0 r[1]=u.rowid
10 RowId 1 3 0 0 r[3]=p.rowid
11 Integer 1 4 0 0 r[4]=1
12 Add 3 4 2 0 r[2]=r[3]+r[4]
13 Ne 1 2 16 0 if r[1]!=r[2] goto 16
14 Column 0 9 5 0 r[5]=u.age
15 ResultRow 5 1 0 0 output=r[5]
16 NextAsync 1 0 0 0
17 NextAwait 1 8 0 0
18 NextAsync 0 0 0 0
19 NextAwait 0 6 0 0
20 Halt 0 0 0 0
21 Transaction 0 0 0 0
22 Goto 0 1 0 0
limbo> SELECT u.age FROM users u JOIN products p ON u.id = p.id + 1;
37
18
33
15
89
24
63
77
13
22
18
```
This whole thing is a bit ad-hoc / "something that works" kind of thing, probably me and @benclmnt need to put our noses down into some books and start to actually build some sort of graph-based query planner after this...
Closes #247
2024-08-01 18:54:33 +03:00
jussisaurio
02cae50324
use move_to() in btree_seek_rowid()
2024-08-01 18:32:01 +03:00
jussisaurio
d5ade427c2
add test for seekrowid order independence
2024-08-01 17:58:38 +03:00
jussisaurio
f344e07868
extract method
2024-08-01 17:54:29 +03:00
jussisaurio
7e88ad64da
Rename and comment
2024-08-01 17:53:21 +03:00
jussisaurio
f46b13690f
augment comment
2024-08-01 17:47:13 +03:00
jussisaurio
6860329940
rename and add comments
2024-08-01 17:45:40 +03:00
jussisaurio
81c2f2eca6
Add comment
2024-08-01 17:40:39 +03:00
jussisaurio
583fe31667
rebase fix
2024-08-01 17:26:00 +03:00
jussisaurio
551b11303f
Broaden the type of expr that qualifies as seekrowid candidate
2024-08-01 17:23:59 +03:00
jussisaurio
64f7e48f1b
Cleanup
2024-08-01 17:23:59 +03:00
jussisaurio
8feb443048
Use SeekRowid instruction on expr1 = expr2 constraints if they contain primary keys
2024-08-01 17:23:59 +03:00
jussisaurio
97dfae437c
SeekRowid VM instruction
2024-08-01 17:23:59 +03:00
jussisaurio
d965998cdf
btree_seek_rowid() implementation
2024-08-01 17:23:59 +03:00
Pekka Enberg
bcd4139a07
Merge 'Storage module cleanup' from Pekka Enberg
...
This pull request moves bunch of code under a new module. The
idea is that we have three major subsystems:
- translate -- that performs code generation for SQL statements
- vdbe -- the runtime that executes SQL statements
- storage -- the storage layer that runtime uses
Closes #254
2024-08-01 12:10:59 +03:00
Pekka Enberg
0bf12ec1b3
core: Move buffer_pool.rs to storage module
2024-08-01 11:53:14 +03:00
Pekka Enberg
ed1c23bfe6
core: Move wal.rs to storage module
2024-08-01 11:52:50 +03:00
Pekka Enberg
f8a43361db
core: Move pager.rs to storage module
2024-08-01 11:52:50 +03:00
Pekka Enberg
7abc48303f
core: Move btree.rs to storage module
2024-08-01 11:52:50 +03:00
Pekka Enberg
307cd71b11
core: Rename storage.rs to storage/mod.rs
...
Prepare for moving more code under storage module.
2024-08-01 11:52:50 +03:00
Pekka Enberg
957cc383c8
core: Fix module documentation format in sqlite3_ondisk.rs
2024-08-01 11:52:18 +03:00
Pekka Enberg
5a7db98efb
Merge 'Initial pass on WAL reader' from Pekka Enberg
...
Closes #249
2024-08-01 11:35:55 +03:00
Pekka Enberg
73bdf1671f
Initial pass on WAL reader
...
These are mostly just stubs for now, but at least we have some code in
place as reminder what we need.
2024-08-01 11:31:17 +03:00
Pekka Enberg
d75817998d
Add merge-pr.py helper script
...
This adds a helper script to generate merge commits that are nicer than
the default Github one.
2024-08-01 10:23:06 +03:00
Pekka Enberg
e88e57f9ba
Merge 'Random clippy cleanups' from Pekka Enberg
...
Closes #253
2024-08-01 10:22:25 +03:00
Pekka Enberg
8c474870c1
core: Eliminate redundant casts
2024-08-01 09:25:25 +03:00
Pekka Enberg
531bf9f96f
Merge pull request #252 from sonhmai/feat/158-partial-support-function-time
...
feat: add time() scalar function partial support without modifier #158
2024-08-01 09:22:08 +03:00
Pekka Enberg
82ff5b9c9b
core: Remove useless use of format!()
2024-08-01 09:16:02 +03:00
Pekka Enberg
6a62e03a7d
simulator: Silence unused variable warning
2024-08-01 09:12:53 +03:00
sonhmai
789ae4becf
feat: add time() scalar function partial support without modifier #158
2024-08-01 13:06:07 +07:00
Pekka Enberg
ce7f373add
Update README.md and CHANGELOG.md
2024-07-31 20:08:46 +03:00
Pekka Enberg
83a14fb6db
Merge pull request #229 from pereman2/ww
...
core: write path
2024-07-31 20:04:06 +03:00
Pekka Enberg
8f6a2fc814
core: Fix I/O compliation on Windows
...
...just steal the generic implementation of pread() and pwrite().
2024-07-31 19:58:37 +03:00
Pekka Enberg
e00690bf9b
core: Fix I/O build on Darwin
2024-07-31 19:52:59 +03:00
Pere Diaz Bou
b8e08dcdc4
core: more2 fix completion darwin io
2024-07-31 18:07:22 +02:00
Pere Diaz Bou
10da6a673d
core: more fix completion darwin io
2024-07-31 18:04:26 +02:00
Pere Diaz Bou
38c407a286
core: fix completion darwin io
2024-07-31 18:01:38 +02:00
Pere Diaz Bou
82ee0e4a00
core: fix completion generic io
2024-07-31 17:58:45 +02:00
Pere Diaz Bou
2b221d2b3c
fix conflicts
...
Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com >
2024-07-31 17:38:33 +02:00
Pere Diaz Bou
8810a5c11e
core: fix move_to replace cursor current page
...
Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com >
2024-07-31 17:27:02 +02:00
Pere Diaz Bou
8c654adc50
core: fix propagation of key to top
...
Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com >
2024-07-31 17:27:02 +02:00
