aljaz/turso
1
0
Fork 0
mirror of https://github.com/aljazceru/turso.git synced 2025-12-28 21:44:21 +01:00
Code Issues Packages Projects Releases Wiki Activity
703 Commits 1 Branch 0 Tags
0e7bd95e4e20046a1e2488e356830fefd1d52d4b
Commit Graph

703 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
sonhmai
0e7bd95e4e core: fix clippy 2024-08-03 20:14:26 +07:00
Pekka Enberg
7937c165fe Merge 'Storage layer cleanups' from Pekka Enberg 
Closes #265
 2024-08-03 11:00:33 +03:00
Pekka Enberg
18883b5a7e core: Document top-level storage module 2024-08-03 10:44:19 +03:00
Pekka Enberg
090a577dd5 core: Move DatabaseStorage to storage/database.rs 2024-08-03 10:41:10 +03:00
Pekka Enberg
83650a797a core: Document DatabaseStorage purpose 2024-08-03 10:37:41 +03:00
Pekka Enberg
3f7c788e5b core: Rename DatabaseStorage methods 
Let's call them read_page() and write_page().
 2024-08-03 10:35:14 +03:00
Pekka Enberg
8a54e31803 core: Rename PageIO to DatabaseStorage 2024-08-03 10:33:52 +03:00
Pekka Enberg
4349b946e5 core: Eliminate PageSource wrapper 
The PageSource wrapper is useless. Let's inline it and use PageIO
directly.
 2024-08-03 10:27:20 +03:00
Pekka Enberg
90308defb2 Merge 'Fix serial type decoding for BLOB and TEXT' from Lauri Virtanen 
Replace `>` with `>=` to match the [SQLite documentation](https://www.sqlite.org/fileformat.html#record_format).

Serial type value `12` is for 0-size BLOB, and `13` is for 0-size TEXT.

Closes #264
 2024-08-02 22:25:10 +03:00
Lauri Virtanen
145af04c7e Fix serial type decoding for BLOB and TEXT 
Replace `>` with `>=` to match the SQLite documentation. Serial type
value `12` is for 0-size BLOB, and `13` is for 0-size TEXT.
 2024-08-02 21:50:50 +03:00
Pekka Enberg
34ed11e3a4 Merge 'Function cleanups' from Pekka Enberg 
Closes #262
 2024-08-02 17:38:53 +03:00
Pekka Enberg
1e9384accc Merge 'update cargo-dist and enable PS installer/github attestations' from ashley williams 
saw you were building a windows bin but not a ps installer and thought i would turn it off. also turned on github attestations because why not.

`cargo dist plan` is basically the same as before but now also produces the PS installer.
```
announcing v0.0.3
  limbo 0.0.3
    source.tar.gz
      [checksum] source.tar.gz.sha256
    limbo-installer.sh
    limbo-installer.ps1
    limbo-aarch64-apple-darwin-update
    limbo-aarch64-apple-darwin.tar.xz
      [bin] limbo
      [misc] CHANGELOG.md, LICENSE.md, README.md
      [checksum] limbo-aarch64-apple-darwin.tar.xz.sha256
    limbo-x86_64-apple-darwin-update
    limbo-x86_64-apple-darwin.tar.xz
      [bin] limbo
      [misc] CHANGELOG.md, LICENSE.md, README.md
      [checksum] limbo-x86_64-apple-darwin.tar.xz.sha256
    limbo-x86_64-pc-windows-msvc-update
    limbo-x86_64-pc-windows-msvc.zip
      [bin] limbo.exe
      [misc] CHANGELOG.md, LICENSE.md, README.md
      [checksum] limbo-x86_64-pc-windows-msvc.zip.sha256
    limbo-x86_64-unknown-linux-gnu-update
    limbo-x86_64-unknown-linux-gnu.tar.xz
      [bin] limbo
      [misc] CHANGELOG.md, LICENSE.md, README.md
      [checksum] limbo-x86_64-unknown-linux-gnu.tar.xz.sha256
  ```

Closes #263
 2024-08-02 17:36:13 +03:00
Pekka Enberg
0affdada2a core: Move datetime.rs to vdbe/ 
The file contains SQL functions invoked by the VDBE so let's move the
file there.
 2024-08-02 17:34:10 +03:00
Pekka Enberg
9c479734be core: Rename to exec_time() 
Follow the same naming convention as other SQL functions.
 2024-08-02 17:34:10 +03:00
Pekka Enberg
465dfa3cb5 core: Rename to exec_date() 
Follow the same naming convention as other SQL functions.
 2024-08-02 17:34:10 +03:00
Ashley Williams
c9c96f01f8 feat(dist): enable github attestations 2024-08-02 09:30:45 -05:00
Ashley Williams
88964e691a feat(dist): update and add PS installer 2024-08-02 09:25:03 -05:00
Pekka Enberg
e926ade455 Merge 'Date and time code cleanups' from Pekka Enberg 
Closes #261
 2024-08-02 16:21:28 +03:00
Pekka Enberg
69eb851120 core/datetime: Move get_max_datetime_exclusive() to bottom 
Move the function so that it's callers are above the function definition
for smoother flow when reading the code.
 2024-08-02 16:15:22 +03:00
Pekka Enberg
6ffb03216f core/datetime: Simplify error handling 2024-08-02 16:15:22 +03:00
Pekka Enberg
3412b65b8a core/datetime: Remove TimeUnit 
...it's not used for anything much.
 2024-08-02 16:04:09 +03:00
Pekka Enberg
763bf17c9e core/datetime: Use "cfg(test)" annotation for tests 2024-08-02 16:03:53 +03:00
Pekka Enberg
f8492c85ae core/datetime: Remove trace calls 
We should trace in high-level code like VDBE interpreter loop, but not
in error handling path of specific SQL functions.
 2024-08-02 16:03:46 +03:00
Pekka Enberg
707d1a0705 Merge 'Update Flake and add Darwin link flags' from Ethan Niser 
This pull request:

 - updates the nix flake to include tcl 8.6 addressing #243

 - updates the nix flake to add the darwin "core foundation" lib

 - updates the make flake to link the core foundation lib on darwin

I was previously unable to run `make test-sqlite3` with this error:
https://gist.github.com/ethanniser/5b96888637da6ea9e9d56dcc3af9b573

But now it is able to run.

Closes #259
 2024-08-02 08:12:14 +03:00
Ethan Niser
d47d494865 update flake to include tcl latest (addresses #243) and add core foundation lib and link flags for darwin 2024-08-01 18:00:46 -07:00
Pekka Enberg
e18fc511bb scripts/merge-pr.py: Use temporal file for commit message 
...fixes problems like eating perfectly valid Markdown and stuff from
pull request descriptions.
 2024-08-01 21:07:39 +03:00
Pekka Enberg
64738d6348 Merge 'Add ability to annotate instructions with comments' from Jussi Saurio 
Similar to what SQLite does.

```
limbo> EXPLAIN SELECT u.age, p.name FROM users u LEFT JOIN products p ON u.first_name = p.name;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     25    0                    0   Start at 25
1     OpenReadAsync      0     2     0                    0   table=u, root=2
2     OpenReadAwait      0     0     0                    0
3     OpenReadAsync      1     3     0                    0   table=p, root=3
4     OpenReadAwait      0     0     0                    0
5     RewindAsync        0     0     0                    0
6     RewindAwait        0     -5    0                    0   Rewind table u
7       Integer          0     1     0                    0   r[1]=0; init LEFT JOIN match flag
8       RewindAsync      1     0     0                    0
9       RewindAwait      1     -11   0                    0   Rewind table p
10        Column         0     1     2                    0   r[2]=u.first_name
11        Column         1     1     3                    0   r[3]=p.name
12        Ne             2     3     17                   0   if r[2]!=r[3] goto 17
13        Integer        1     1     0                    0   r[1]=1; record LEFT JOIN hit
14        Column         0     9     4                    0   r[4]=u.age
15        Column         1     1     5                    0   r[5]=p.name
16        ResultRow      4     2     0                    0   output=r[4..5]
17      NextAsync        1     0     0                    0
18      NextAwait        1     9     0                    0
19      IfPos            1     22    0                    0   r[1]>0 -> r[1]-=0, goto 22
20      NullRow          1     0     0                    0   Set cursor 1 to a (pseudo) NULL row
21      Goto             0     13    0                    0
22    NextAsync          0     0     0                    0
23    NextAwait          0     6     0                    0
24    Halt               0     0     0                    0
25    Transaction        0     0     0                    0
26    Goto               0     1     0                    0
```

Closes #258
 2024-08-01 21:07:31 +03:00
jussisaurio
20c085614f Add ability to annotate instructions with comments 2024-08-01 20:34:45 +03:00
Pekka Enberg
fedb558346 Update COMPAT.md 2024-08-01 20:24:44 +03:00
Pekka Enberg
5d29e04854 Update COMPAT.md 2024-08-01 19:24:10 +03:00
Pekka Enberg
cb34deef09 Limbo 0.0.3 2024-08-01 18:57:13 +03:00
Pekka Enberg
6af9aafddc Update CHANGELOG.md 2024-08-01 18:57:13 +03:00
Pekka Enberg
d97a434e29 Merge 'Use SeekRowId to avoid nested scans' from Jussi Saurio 
The SQLite `SeekRowid` instruction tries to find a record in a B-tree table with a particular `rowid`  (or INTEGER PRIMARY KEY, if defined), and jumps if it is not found. In this PR, constraints like `tbl.id = 5` or `tbl.id = tbl2.id` are transformed into special `SeekRowid` expressions that emit `SeekRowid` VM instructions. This avoids a table scan and instead completes in `log N` time (for a table of `N` rows), because of how B-Trees work.


`LoopInfo` now contains a `Plan` which is either `Scan` or `SeekRowid` -- in the case of `SeekRowid` no `Rewind`/`Next` instructions are emitted - i.e. no looping is done.

`BTreeCursor` now implements a `btree_seek_rowid()` method that tries to find a row by `rowid`.

Our loop order is currently static, i.e. `SELECT * from a join b join c` always results in "loop a, loop b, loop c", so `SeekRowId` is only supported for equality expressions where the non-PK side of the expression only refers to outer loops or constants. Examples:

**Because `u.id` refers to an outer loop compared to the primary key `p.id`, `p` is selected for SeekRowid optimization:**
```
limbo> explain SELECT u.age FROM users u JOIN products p ON u.id = p.id

addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     14    0                    0   Start at 14
1     OpenReadAsync      0     2     0                    0   table=u, root=2
2     OpenReadAwait      0     0     0                    0   
3     OpenReadAsync      1     3     0                    0   table=p, root=3
4     OpenReadAwait      0     0     0                    0   
5     RewindAsync        0     0     0                    0   
6     RewindAwait        0     -5    0                    0   Rewind table u
7       RowId            0     1     0                    0   r[1]=u.rowid
8       SeekRowid        1     1     11                   0   if (r[1]!=p.rowid) goto 11
9       Column           0     9     2                    0   r[2]=u.age
10      ResultRow        2     1     0                    0   output=r[2]
11    NextAsync          0     0     0                    0   
12    NextAwait          0     6     0                    0   
13    Halt               0     0     0                    0   
14    Transaction        0     0     0                    0   
15    Goto               0     1     0                    0

limbo> SELECT u.age FROM users u JOIN products p ON u.id = p.id
94
37
18
33
15
89
24
63
77
13
22
```

**Because `5` refers to a constant and `u.id` is a primary key, `u` is selected for SeekRowid optimization:**
```
limbo> explain SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = 5;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     15    0                    0   Start at 15
1     OpenReadAsync      0     2     0                    0   table=u, root=2
2     OpenReadAwait      0     0     0                    0   
3     OpenReadAsync      1     3     0                    0   table=p, root=3
4     OpenReadAwait      0     0     0                    0   
5     Integer            5     1     0                    0   r[1]=5
6     SeekRowid          0     1     14                   0   if (r[1]!=u.rowid) goto 14
7     RewindAsync        1     0     0                    0   
8     RewindAwait        1     -8    0                    0   Rewind table p
9       Column           0     1     2                    0   r[2]=u.first_name
10      Column           1     1     3                    0   r[3]=p.name
11      ResultRow        2     2     0                    0   output=r[2..3]
12    NextAsync          1     0     0                    0   
13    NextAwait          1     8     0                    0   
14    Halt               0     0     0                    0   
15    Transaction        0     0     0                    0   
16    Goto               0     1     0                    0   
limbo> SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = 5;
Edward|hat
Edward|cap
Edward|shirt
Edward|sweater
Edward|sweatshirt
Edward|shorts
Edward|jeans
Edward|sneakers
Edward|boots
Edward|coat
Edward|accessories
```

**Same, but LEFT JOIN:**
```
limbo> EXPLAIN SELECT u.first_name, p.name FROM users u LEFT JOIN products p ON u.id = 5;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     20    0                    0   Start at 20
1     OpenReadAsync      0     2     0                    0   table=u, root=2
2     OpenReadAwait      0     0     0                    0   
3     OpenReadAsync      1     3     0                    0   table=p, root=3
4     OpenReadAwait      0     0     0                    0   
5     Integer            0     1     0                    0   r[1]=0
6     RewindAsync        1     0     0                    0   
7     RewindAwait        1     -11   0                    0   Rewind table p
8       Integer          5     2     0                    0   r[2]=5
9       SeekRowid        0     2     14                   0   if (r[2]!=u.rowid) goto 14
10      Integer          1     1     0                    0   r[1]=1
11      Column           0     1     3                    0   r[3]=u.first_name
12      Column           1     1     4                    0   r[4]=p.name
13      ResultRow        3     2     0                    0   output=r[3..4]
14    NextAsync          1     0     0                    0   
15    NextAwait          1     7     0                    0   
16    IfPos              1     19    0                    0   r[1]>0 -> r[1]-=0, goto 19
17    NullRow            1     0     0                    0   Set cursor 1 to a (pseudo) NULL row
18    Goto               0     10    0                    0   
19    Halt               0     0     0                    0   
20    Transaction        0     0     0                    0   
21    Goto               0     1     0                    0   
limbo> SELECT u.first_name, p.name FROM users u LEFT JOIN products p ON u.id = 5;
Edward|hat
Edward|cap
Edward|shirt
Edward|sweater
Edward|sweatshirt
Edward|shorts
Edward|jeans
Edward|sneakers
Edward|boots
Edward|coat
Edward|accessories
```

**Both `p` and `u` selected for optimization:**
```
limbo> EXPLAIN SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = p.id and u.id = 5;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     13    0                    0   Start at 13
1     OpenReadAsync      0     2     0                    0   table=u, root=2
2     OpenReadAwait      0     0     0                    0   
3     OpenReadAsync      1     3     0                    0   table=p, root=3
4     OpenReadAwait      0     0     0                    0   
5     Integer            5     1     0                    0   r[1]=5
6     SeekRowid          0     1     12                   0   if (r[1]!=u.rowid) goto 12
7     RowId              0     2     0                    0   r[2]=u.rowid
8     SeekRowid          1     2     12                   0   if (r[2]!=p.rowid) goto 12
9     Column             0     1     3                    0   r[3]=u.first_name
10    Column             1     1     4                    0   r[4]=p.name
11    ResultRow          3     2     0                    0   output=r[3..4]
12    Halt               0     0     0                    0   
13    Transaction        0     0     0                    0   
14    Goto               0     1     0                    0   

limbo> SELECT u.first_name, p.name FROM users u JOIN products p ON u.id = p.id and u.id = 5;
Edward|sweatshirt
```

**`p.id + 1` refers to an INNER loop compared to the primary key `u.id`, so optimization is skipped:**
```
limbo> EXPLAIN SELECT u.age FROM users u JOIN products p ON u.id = p.id + 1;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     21    0                    0   Start at 21
1     OpenReadAsync      0     2     0                    0   table=u, root=2
2     OpenReadAwait      0     0     0                    0   
3     OpenReadAsync      1     3     0                    0   table=p, root=3
4     OpenReadAwait      0     0     0                    0   
5     RewindAsync        0     0     0                    0   
6     RewindAwait        0     -5    0                    0   Rewind table u
7       RewindAsync      1     0     0                    0   
8       RewindAwait      1     -8    0                    0   Rewind table p
9         RowId          0     1     0                    0   r[1]=u.rowid
10        RowId          1     3     0                    0   r[3]=p.rowid
11        Integer        1     4     0                    0   r[4]=1
12        Add            3     4     2                    0   r[2]=r[3]+r[4]
13        Ne             1     2     16                   0   if r[1]!=r[2] goto 16
14        Column         0     9     5                    0   r[5]=u.age
15        ResultRow      5     1     0                    0   output=r[5]
16      NextAsync        1     0     0                    0   
17      NextAwait        1     8     0                    0   
18    NextAsync          0     0     0                    0   
19    NextAwait          0     6     0                    0   
20    Halt               0     0     0                    0   
21    Transaction        0     0     0                    0   
22    Goto               0     1     0                    0   
limbo> SELECT u.age FROM users u JOIN products p ON u.id = p.id + 1;
37
18
33
15
89
24
63
77
13
22
18
```

This whole thing is a bit ad-hoc / "something that works" kind of thing, probably me and @benclmnt need to put our noses down into some books and start to actually build some sort of graph-based query planner after this...

Closes #247
 2024-08-01 18:54:33 +03:00
jussisaurio
02cae50324 use move_to() in btree_seek_rowid() 2024-08-01 18:32:01 +03:00
jussisaurio
d5ade427c2 add test for seekrowid order independence 2024-08-01 17:58:38 +03:00
jussisaurio
f344e07868 extract method 2024-08-01 17:54:29 +03:00
jussisaurio
7e88ad64da Rename and comment 2024-08-01 17:53:21 +03:00
jussisaurio
f46b13690f augment comment 2024-08-01 17:47:13 +03:00
jussisaurio
6860329940 rename and add comments 2024-08-01 17:45:40 +03:00
jussisaurio
81c2f2eca6 Add comment 2024-08-01 17:40:39 +03:00
jussisaurio
583fe31667 rebase fix 2024-08-01 17:26:00 +03:00
jussisaurio
551b11303f Broaden the type of expr that qualifies as seekrowid candidate 2024-08-01 17:23:59 +03:00
jussisaurio
64f7e48f1b Cleanup 2024-08-01 17:23:59 +03:00
jussisaurio
8feb443048 Use SeekRowid instruction on expr1 = expr2 constraints if they contain primary keys 2024-08-01 17:23:59 +03:00
jussisaurio
97dfae437c SeekRowid VM instruction 2024-08-01 17:23:59 +03:00
jussisaurio
d965998cdf btree_seek_rowid() implementation 2024-08-01 17:23:59 +03:00
Pekka Enberg
bcd4139a07 Merge 'Storage module cleanup' from Pekka Enberg 
This pull request moves bunch of code under a new  module. The
idea is that we have three major subsystems:

- translate -- that performs code generation for SQL statements
- vdbe -- the runtime that executes SQL statements
- storage -- the storage layer that runtime uses

Closes #254
 2024-08-01 12:10:59 +03:00
Pekka Enberg
0bf12ec1b3 core: Move buffer_pool.rs to storage module 2024-08-01 11:53:14 +03:00
Pekka Enberg
ed1c23bfe6 core: Move wal.rs to storage module 2024-08-01 11:52:50 +03:00
Pekka Enberg
f8a43361db core: Move pager.rs to storage module 2024-08-01 11:52:50 +03:00