aljaz/turso
1
0
Fork 0
mirror of https://github.com/aljazceru/turso.git synced 2026-01-01 07:24:19 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,002 Commits 1 Branch 0 Tags
fdf3b0d16bcdc9bed7b46ddf41aa8441558df8ae
Commit Graph

1002 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pekka Enberg
fdf3b0d16b Merge '(core): Primary key index scans and single-column secondary index scans' from Jussi Saurio 
This PR adds an index on `users.age` to `testing.db`, and support for
indexed lookups. Only single-column ascending indexes are currently
supported.
This PR also gets rid of `Operator::Seekrowid` in favor of
`Operator::Search` which handles all non-full-table-scan searches: 1.
integer primary key (rowid) point queries 2. integer primary key index
scans, and 3. secondary index scans.
examples:
```
limbo> select first_name, age from users where age > 90 limit 10;
Miranda|90
Sarah|90
Justin|90
Justin|90
John|90
Jeremy|90
Stephanie|90
Joshua|90
Jenny|90
Jennifer|90

limbo> explain query plan select first_name, age from users where age > 90 limit 10;
QUERY PLAN
`--TAKE 10
   `--PROJECT first_name, age
   |  `--SEARCH users USING INDEX age_idx

limbo> explain select first_name, age from users where age > 90 limit 10;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     15    0                    0   Start at 15
1     OpenReadAsync      0     2     0                    0   table=users, root=2
2     OpenReadAwait      0     0     0                    0
3     OpenReadAsync      1     274   0                    0   table=age_idx, root=274
4     OpenReadAwait      0     0     0                    0
5     Integer            90    1     0                    0   r[1]=90
6     SeekGT             1     14    1                    0
7       DeferredSeek     1     0     0                    0
8       Column           0     1     2                    0   r[2]=users.first_name
9       Column           0     9     3                    0   r[3]=users.age
10      ResultRow        2     2     0                    0   output=r[2..3]
11      DecrJumpZero     4     14    0                    0   if (--r[4]==0) goto 14
12    NextAsync          1     0     0                    0
13    NextAwait          1     7     0                    0
14    Halt               0     0     0                    0
15    Transaction        0     0     0                    0
16    Integer            10    4     0                    0   r[4]=10
17    Goto               0     1     0                    0
```
Sqlite version:
```
sqlite> explain select first_name, age from users where age > 90 limit 10;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     13    0                    0   Start at 13
1     Integer        10    1     0                    0   r[1]=10; LIMIT counter
2     OpenRead       0     2     0     10             0   root=2 iDb=0; users
3     OpenRead       1     274   0     k(2,,)         0   root=274 iDb=0; age_idx
4     Integer        90    2     0                    0   r[2]=90
5     SeekGT         1     12    2     1              0   key=r[2]
6       DeferredSeek   1     0     0                    0   Move 0 to 1.rowid if needed
7       Column         0     1     3                    0   r[3]= cursor 0 column 1
8       Column         1     0     4                    0   r[4]= cursor 1 column 0
9       ResultRow      3     2     0                    0   output=r[3..4]
10      DecrJumpZero   1     12    0                    0   if (--r[1])==0 goto 12
11    Next           1     6     0                    0
12    Halt           0     0     0                    0
13    Transaction    0     0     3     0              1   usesStmtJournal=0
14    Goto           0     1     0                    0
```
---
´Seek` instructions are also now supported for primary key rowid
searches:
```
limbo> select id, first_name from users where id > 9995;
9996|Donald
9997|Ruth
9998|Dorothy
9999|Gina
10000|Nicole

limbo> explain query plan select id, first_name from users where id > 9995;
QUERY PLAN
`--PROJECT id, first_name
   `--SEARCH users USING INTEGER PRIMARY KEY (rowid=?)

limbo> explain select id, first_name from users where id > 9995;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     11    0                    0   Start at 11
1     OpenReadAsync      0     2     0                    0   table=users, root=2
2     OpenReadAwait      0     0     0                    0
3     Integer            9995  1     0                    0   r[1]=9995
4     SeekGT             0     10    1                    0
5       RowId            0     2     0                    0   r[2]=users.rowid
6       Column           0     1     3                    0   r[3]=users.first_name
7       ResultRow        2     2     0                    0   output=r[2..3]
8     NextAsync          0     0     0                    0
9     NextAwait          0     5     0                    0
10    Halt               0     0     0                    0
11    Transaction        0     0     0                    0
12    Goto               0     1     0                    0
```
sqlite:
```
sqlite> explain select id, first_name from users where id > 9995;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     8     0                    0   Start at 8
1     OpenRead       0     2     0     2              0   root=2 iDb=0; users
2     SeekGT         0     7     1                    0   key=r[1]; pk
3       Rowid          0     2     0                    0   r[2]=users.rowid
4       Column         0     1     3                    0   r[3]= cursor 0 column 1
5       ResultRow      2     2     0                    0   output=r[2..3]
6     Next           0     3     0                    0
7     Halt           0     0     0                    0
8     Transaction    0     0     3     0              1   usesStmtJournal=0
9     Integer        9995  1     0                    0   r[1]=9995
10    Goto           0     1     0                    0
```
---
More complex example with a join that uses both a rowid lookup and a
secondary index scan:
```
limbo> explain query plan select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
QUERY PLAN
`--PROJECT u.first_name, p.name
   `--JOIN
   |  |--SEARCH u USING INDEX age_idx
   |  `--SEARCH p USING INTEGER PRIMARY KEY (rowid=?)

limbo> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     18    0                    0   Start at 18
1     OpenReadAsync      0     2     0                    0   table=u, root=2
2     OpenReadAwait      0     0     0                    0
3     OpenReadAsync      1     274   0                    0   table=age_idx, root=274
4     OpenReadAwait      0     0     0                    0
5     OpenReadAsync      2     3     0                    0   table=p, root=3
6     OpenReadAwait      0     0     0                    0
7     Integer            70    1     0                    0   r[1]=70
8     SeekGT             1     17    1                    0
9       DeferredSeek     1     0     0                    0
10      RowId            0     2     0                    0   r[2]=u.rowid
11      SeekRowid        2     2     15                   0   if (r[2]!=p.rowid) goto 15
12      Column           0     1     3                    0   r[3]=u.first_name
13      Column           2     1     4                    0   r[4]=p.name
14      ResultRow        3     2     0                    0   output=r[3..4]
15    NextAsync          1     0     0                    0
16    NextAwait          1     9     0                    0
17    Halt               0     0     0                    0
18    Transaction        0     0     0                    0
19    Goto               0     1     0                    0
```
sqlite:
```
sqlite> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     14    0                    0   Start at 14
1     OpenRead       0     2     0     10             0   root=2 iDb=0; users
2     OpenRead       2     274   0     k(2,,)         0   root=274 iDb=0; age_idx
3     OpenRead       1     3     0     2              0   root=3 iDb=0; products
4     Integer        70    1     0                    0   r[1]=70
5     SeekGT         2     13    1     1              0   key=r[1]
6       DeferredSeek   2     0     0                    0   Move 0 to 2.rowid if needed
7       IdxRowid       2     2     0                    0   r[2]=rowid; users.rowid
8       SeekRowid      1     12    2                    0   intkey=r[2]
9       Column         0     1     3                    0   r[3]= cursor 0 column 1
10      Column         1     1     4                    0   r[4]= cursor 1 column 1
11      ResultRow      3     2     0                    0   output=r[3..4]
12    Next           2     6     0                    0
13    Halt           0     0     0                    0
14    Transaction    0     0     3     0              1   usesStmtJournal=0
15    Goto           0     1     0                    0
```

Closes #350
 2024-10-13 10:11:23 +03:00
jussisaurio
c58b7b42c0 Merge 'Add support for Python 3.13 and drop support for Python 3.8' from Lauri Virtanen 
The 7th of October 2024 was the first day of Python 3.13 and the last of
Python 3.8 official support.
For reference: https://devguide.python.org/versions/

Closes #367
 2024-10-12 17:24:53 +03:00
Lauri Virtanen
aa526f4cb3 Add support for Python 3.13 and drop support for Python 3.8 2024-10-12 15:16:14 +03:00
Pekka Enberg
87dd6075f1 Merge 'Fix the Python CI builds' from GV 
Fixes the Python CI build.

Closes #363
 2024-10-10 08:58:17 +03:00
gandeevanr
be94138a5a Add coverage to development dependencies in pyproject.toml and requirements-dev.txt. 2024-10-09 19:00:53 -07:00
Pekka Enberg
d6829e9794 Merge '[sorter] Hold records in Vec instead of a BTreeMap' from Arpit Saxena 
We now just insert them one after the other in the vector. When rewind
is called, the vector is sorted. Iterating is just taking elements from
the vector.
Related to #191
SQLite3:
```sh
$ time sqlite3 testing/testing.db "SELECT id FROM users ORDER BY zipcode" > /dev/null

real    0m0.020s
user    0m0.013s
sys     0m0.008s
```
Limbo without this PR:
```sh
$ time target/release/limbo testing/testing.db "SELECT id FROM users ORDER BY zipcode" > /dev/null

real    0m0.285s
user    0m0.257s
sys     0m0.014s
```
Limbo with this PR:
```sh
$ time target/release/limbo testing/testing.db "SELECT id FROM users ORDER BY zipcode" > /dev/null

real    0m0.084s
user    0m0.043s
sys     0m0.032s
```

Closes #362
 2024-10-09 16:32:48 +03:00
Arpit Saxena
28a603f56a [sorter] Hold records in Vec instead of a BTreeMap 
We now just insert them one after the other in the vector. When rewind
is called, the vector is sorted. Iterating is just taking elements from
the vector
 2024-10-09 17:58:28 +05:30
jussisaurio
556f4b73c9 Refine edge case handling: add optional predicate to get_next_record() 2024-10-08 08:23:30 +03:00
jussisaurio
43038cb6aa Handle seek() edge case with index seek 2024-10-08 07:45:21 +03:00
jussisaurio
572db69b5e Add TODO comment about index corner case 2024-10-07 17:12:19 +03:00
jussisaurio
93a8110773 dont assume index key has rowid in the second column: its the last 2024-10-07 17:05:38 +03:00
jussisaurio
8563d620af renaming 2024-10-07 17:03:50 +03:00
jussisaurio
fc71f2b32f traverse index properly 
interior index cells have values that are not in the leaves, e.g.

     (interior: 3)
    /            \
(leaf: 2)     (leaf: 4)

so their values need to be emitted after the left subtree is emitted.
 2024-10-07 13:04:03 +03:00
Pekka Enberg
0d673710e2 Merge 'Add instr(X,Y) scalar function' from Lauri Virtanen 
Relates to issue #144
> ## instr(X,Y)
>
> The instr(X,Y) function finds the first occurrence of string Y within
string X and returns the number of prior characters plus 1, or 0 if Y is
nowhere found within X. Or, if X and Y are both BLOBs, then instr(X,Y)
returns one more than the number bytes prior to the first occurrence of
Y, or 0 if Y does not occur anywhere within X. If both arguments X and Y
to instr(X,Y) are non-NULL and are not BLOBs then both are interpreted
as strings. If either X or Y are NULL in instr(X,Y) then the result is
NULL.

Closes #357
 2024-10-07 08:59:40 +03:00
jussisaurio
e5cf052f07 Why do sqlite btree child keys have <= keys and not < keys 2024-10-06 23:48:59 +03:00
Lauri Virtanen
0ae1412193 Add instr(X,Y) scalar function 
Relates to issue #144
 2024-10-06 20:19:37 +03:00
jussisaurio
15a66ea662 single seek function in cursor trait 2024-10-06 09:21:15 +03:00
jussisaurio
6e7db36121 reorder 2024-10-06 00:58:32 +03:00
jussisaurio
af9a751d36 Single seek function 2024-10-06 00:56:18 +03:00
jussisaurio
1ae8d28669 Use same move_to() function for tables and indexes 2024-10-06 00:51:14 +03:00
jussisaurio
37f877109e Reduce duplication in btree.rs 2024-10-06 00:39:50 +03:00
jussisaurio
bb1c8b65e8 fmt 2024-10-06 00:22:12 +03:00
jussisaurio
dde10d2dd7 Better EXPLAIN QUERY PLAN for Operator::Search 2024-10-06 00:19:56 +03:00
jussisaurio
47534cb8df Get rid of Seekrowid operator in favor of a unified Search operator 2024-10-06 00:11:38 +03:00
jussisaurio
d3e797f59e rewind_labels was renamed to scan_loop_body_labels 2024-10-05 18:27:18 +03:00
jussisaurio
d22dbe9840 remove garbage comment 2024-10-05 18:25:04 +03:00
jussisaurio
3826d4e1ff Add comment about code duplication 2024-10-05 18:25:04 +03:00
jussisaurio
d2233d69d3 Dont assume the rowid is the second column - it's the last 2024-10-05 18:25:04 +03:00
jussisaurio
db0e2ea54f Change another compat test to work around sqlite's weird choice to use the age index 2024-10-05 18:25:04 +03:00
jussisaurio
d8a695a991 rename tests 2024-10-05 18:25:04 +03:00
jussisaurio
43015f6949 Workaround for compat test 2024-10-05 18:25:04 +03:00
jussisaurio
9169f6e39b same thing 2024-10-05 18:25:04 +03:00
jussisaurio
fe90aacd35 Handle CursorResult in deferred seek 2024-10-05 18:25:04 +03:00
jussisaurio
02d6fa31d3 Fix .schema users not displaying indexes on the users table 2024-10-05 18:25:04 +03:00
jussisaurio
3a11887122 fixerinos 2024-10-05 18:25:04 +03:00
jussisaurio
ed19f47762 fix 2024-10-05 18:25:04 +03:00
jussisaurio
ff236c7781 Fix not advancing the cell index of pages 2024-10-05 18:25:04 +03:00
jussisaurio
99871bbeea yield on io 2024-10-05 18:25:04 +03:00
jussisaurio
e118b70127 fmt 2024-10-05 18:25:04 +03:00
jussisaurio
3d56fbd91c stuff 2024-10-05 18:25:04 +03:00
jussisaurio
f02da18acd index scan wip foo doesnt work yet 2024-10-05 18:25:04 +03:00
jussisaurio
d3015ad854 Merge 'Fix NextAwait's next instruction; rename rewind_labels' from Arpit Saxena 
Closes #351
NextAwait's next instruction was pointing to RewindAwait earlier which
is not current. The instruction now points to the instruction after
RewindAwait, which will be the loop body. Hence, rename rewind_labels to
scan_loop_body_labels to make it clearer

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #354
 2024-10-05 18:23:37 +03:00
Pekka Enberg
765870ce69 Update CHANGELOG.md 2024-10-03 10:35:56 +03:00
Pekka Enberg
fd6c27ab72 Merge 'Add randomblob(N) scalar function' from Lauri Virtanen 
Relates to issue #144

Closes #355
 2024-10-03 10:35:23 +03:00
Lauri Virtanen
9e80a0c4a8 Add randomblob(N) scalar function 
Relates to issue #144
 2024-10-03 00:05:23 +03:00
Pekka Enberg
b7656c265c Fix source formatting with cargo fmt... 2024-10-02 11:06:38 +03:00
Pekka Enberg
6fcd818160 Merge 'Add unhex(X) scalar function' from Lauri Virtanen 
This commit adds `unhex(X)` scalar function. Function with `unhex(X,Y)`
two arguments is not implemented yet.
Relates to issue #144

Closes #353
 2024-10-02 11:01:15 +03:00
Pekka Enberg
b2fd509ecc Merge 'Add zeroblob(N) scalar function' from Lauri Virtanen 
Relates to issue #144

Closes #352
 2024-10-02 10:59:22 +03:00
Arpit Saxena
47c8fd1964 Fix NextAwait's next instruction; rename rewind_labels 
Closes #351

NextAwait's next instruction was pointing to RewindAwait earlier which
is not current. The instruction now points to the instruction after
RewindAwait, which will be the loop body. Hence, rename rewind_labels to
scan_loop_body_labels to make it clearer
 2024-10-02 11:58:42 +05:30
Lauri Virtanen
adc6f9b6cd Add unhex(X) scalar function 
This commit adds `unhex(X)` scalar function. Function with `unhex(X,Y)`
two arguments is not implemented yet.

Relates to issue #144
 2024-09-30 00:13:43 +03:00