Jussi Saurio
d286a56e15
refactor: fold Async/Await insns into a single instruction
2025-04-14 09:40:20 +03:00
Jussi Saurio
cbe3500b7e
Merge 'Code clean-ups' from Diego Reis
...
While developing I found that some things could be improved :)
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1325
2025-04-13 17:02:12 +03:00
Diego Reis
51eb2af06a
core(refactor): Add CreateBTreeFlags
...
Passing 1s and 0s with comments is not rustacean, and since we already follow the pattern of struct flags in other sections of the codebase it's better use it here too.
2025-04-13 01:46:30 -03:00
Diego Reis
db0f07499d
core/translate: Fix naive comparison between Binary expressions during register optimization
2025-04-12 16:39:30 -03:00
Jussi Saurio
6bea4de30f
Check that index seek key members are not null
2025-04-11 17:22:46 +03:00
Pekka Enberg
e3a4400329
Merge 'Multi column indexes + index seek refactor' from Jussi Saurio
...
# Multi column indexes + index seek refactor
## PR reader guide
I would say mostly you should just focus on the content of
`optimizer.rs` and `plan.rs` because the rest is just small type
changes, or in the case of `main_loop.rs`, a bunch of logic was just
moved out of there and rewritten.
## New feature - multi column index seeks
This PR adds support for utilizing multi-column indexes properly, i.e.
using as many columns in the seek key as possible. Previously, we only
used max one column per index. I've modified the existing compound index
seek fuzz test to use this functionality.
## Refactoring of index seek related logic
This PR moves a lot of index seek related logic out of `main_loop.rs`
into `optimizer.rs` and `plan.rs` and introduces a bunch of helper
structures to model finding and using an index to perform a seek + scan.
## Examples
Here are some examples of multi-column seeks:
### Example table setup:
```sql
sqlite> CREATE TABLE t(a,b,c,d,e);
sqlite> CREATE INDEX abc ON t (a,b,c);
-- create 10000 rows with random values between 0-9 for all columns
sqlite >INSERT INTO t SELECT ABS(RANDOM() % 10),ABS(RANDOM() % 10),ABS(RANDOM() % 10),ABS(RANDOM() % 10),ABS(RANDOM() % 10) FROM generate_series(1,10000,1);
```
### Example bytecode plans, results and timings vs main branch:
```sql
limbo> EXPLAIN SELECT * FROM t WHERE a = 5 and b = 6 and c = 7;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 20 0 0 Start at 20
1 OpenReadAsync 0 2 0 0 table=t, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=abc, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 5 6 0 0 r[6]=5
6 Integer 6 7 0 0 r[7]=6
7 Integer 7 8 0 0 r[8]=7
8 SeekGE 1 19 6 0 key=[6..8]
9 IdxGT 1 19 6 0 key=[6..8]
10 DeferredSeek 1 0 0 0
11 Column 0 0 1 0 r[1]=t.a
12 Column 0 1 2 0 r[2]=t.b
13 Column 0 2 3 0 r[3]=t.c
14 Column 0 3 4 0 r[4]=t.d
15 Column 0 4 5 0 r[5]=t.e
16 ResultRow 1 5 0 0 output=r[1..5]
17 NextAsync 1 0 0 0
18 NextAwait 1 9 0 0
19 Halt 0 0 0 0
20 Transaction 0 0 0 0 write=false
21 Goto 0 1 0 0
limbo> SELECT * FROM t WHERE a = 5 and b = 6 and c = 7;
5|6|7|9|9
5|6|7|4|7
5|6|7|3|2
5|6|7|3|7
5|6|7|5|2
5|6|7|5|3
5|6|7|9|7
runtime (debug build, this branch): total: 2 ms (this includes parsing/coloring of cli app)
runtime (debud build, main branch): total: 67 ms (this includes parsing/coloring of cli app)
```
```sql
limbo> EXPLAIN SELECT * FROM t WHERE a = 5 and b = 6 and c < 7;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 21 0 0 Start at 21
1 OpenReadAsync 0 2 0 0 table=t, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=abc, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 5 6 0 0 r[6]=5
6 Integer 6 7 0 0 r[7]=6
7 Null 0 8 0 0 r[8]=NULL
8 SeekGT 1 20 6 0 key=[6..8]
9 Integer 7 8 0 0 r[8]=7
10 IdxGE 1 20 6 0 key=[6..8]
11 DeferredSeek 1 0 0 0
12 Column 0 0 1 0 r[1]=t.a
13 Column 0 1 2 0 r[2]=t.b
14 Column 0 2 3 0 r[3]=t.c
15 Column 0 3 4 0 r[4]=t.d
16 Column 0 4 5 0 r[5]=t.e
17 ResultRow 1 5 0 0 output=r[1..5]
18 NextAsync 1 0 0 0
19 NextAwait 1 10 0 0
20 Halt 0 0 0 0
21 Transaction 0 0 0 0 write=false
22 Goto 0 1 0 0
limbo> SELECT * FROM t WHERE a = 5 and b = 6 and c < 7;
5|6|0|0|3
5|6|0|5|1
5|6|0|3|1
5|6|0|6|3
5|6|0|8|1
5|6|0|2|7
5|6|0|9|9
5|6|0|5|3
5|6|0|4|2
5|6|0|4|2
5|6|0|0|2
5|6|0|7|2
5|6|1|8|5
5|6|1|7|5
5|6|1|7|2
5|6|1|1|2
5|6|1|6|5
5|6|1|1|5
5|6|1|5|7
5|6|1|1|9
5|6|1|4|3
5|6|1|1|2
5|6|1|2|2
5|6|1|4|4
5|6|1|9|6
5|6|1|2|5
5|6|1|2|4
5|6|1|7|1
5|6|2|0|9
5|6|2|6|9
5|6|2|4|5
5|6|2|9|3
5|6|2|5|2
5|6|2|9|0
5|6|2|7|1
5|6|3|6|5
5|6|3|8|5
5|6|3|5|4
5|6|3|5|2
5|6|3|1|1
5|6|3|2|0
5|6|3|9|3
5|6|3|6|9
5|6|3|7|6
5|6|3|3|5
5|6|3|0|8
5|6|3|6|4
5|6|4|1|1
5|6|4|9|8
5|6|4|3|7
5|6|4|1|3
5|6|4|8|9
5|6|4|9|7
5|6|4|7|9
5|6|4|8|8
5|6|4|3|1
5|6|4|2|6
5|6|4|5|7
5|6|4|2|6
5|6|4|4|3
5|6|5|2|4
5|6|5|6|7
5|6|5|3|8
5|6|5|7|8
5|6|5|9|6
5|6|5|2|7
5|6|5|1|7
5|6|5|0|6
5|6|6|2|4
5|6|6|9|4
5|6|6|4|9
5|6|6|5|6
5|6|6|2|2
5|6|6|0|6
runtime (debug build, this branch): total: 9 ms (this includes parsing/coloring of cli app)
runtime (debug build, main branch): total: 71 ms (this includes parsing/coloring of cli app)
```
```sql
limbo> EXPLAIN SELECT * FROM t WHERE a = 5 and b = 6 and c < 7 ORDER BY a desc, b desc, c desc;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 20 0 0 Start at 20
1 OpenReadAsync 0 2 0 0 table=t, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=abc, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 5 6 0 0 r[6]=5
6 Integer 6 7 0 0 r[7]=6
7 Integer 7 8 0 0 r[8]=7
8 SeekLT 1 19 6 0 key=[6..8]
9 IdxLT 1 19 6 0 key=[6..7]
10 DeferredSeek 1 0 0 0
11 Column 0 0 1 0 r[1]=t.a
12 Column 0 1 2 0 r[2]=t.b
13 Column 0 2 3 0 r[3]=t.c
14 Column 0 3 4 0 r[4]=t.d
15 Column 0 4 5 0 r[5]=t.e
16 ResultRow 1 5 0 0 output=r[1..5]
17 PrevAsync 1 0 0 0
18 PrevAwait 1 0 0 0
19 Halt 0 0 0 0
20 Transaction 0 0 0 0 write=false
21 Goto 0 1 0 0
limbo> SELECT * FROM t WHERE a = 5 and b = 6 and c < 7 ORDER BY a desc, b desc, c desc;
5|6|6|0|6
5|6|6|2|2
5|6|6|5|6
5|6|6|4|9
5|6|6|9|4
5|6|6|2|4
5|6|5|0|6
5|6|5|1|7
5|6|5|2|7
5|6|5|9|6
5|6|5|7|8
5|6|5|3|8
5|6|5|6|7
5|6|5|2|4
5|6|4|4|3
5|6|4|2|6
5|6|4|5|7
5|6|4|2|6
5|6|4|3|1
5|6|4|8|8
5|6|4|7|9
5|6|4|9|7
5|6|4|8|9
5|6|4|1|3
5|6|4|3|7
5|6|4|9|8
5|6|4|1|1
5|6|3|6|4
5|6|3|0|8
5|6|3|3|5
5|6|3|7|6
5|6|3|6|9
5|6|3|9|3
5|6|3|2|0
5|6|3|1|1
5|6|3|5|2
5|6|3|5|4
5|6|3|8|5
5|6|3|6|5
5|6|2|7|1
5|6|2|9|0
5|6|2|5|2
5|6|2|9|3
5|6|2|4|5
5|6|2|6|9
5|6|2|0|9
5|6|1|7|1
5|6|1|2|4
5|6|1|2|5
5|6|1|9|6
5|6|1|4|4
5|6|1|2|2
5|6|1|1|2
5|6|1|4|3
5|6|1|1|9
5|6|1|5|7
5|6|1|1|5
5|6|1|6|5
5|6|1|1|2
5|6|1|7|2
5|6|1|7|5
5|6|1|8|5
5|6|0|7|2
5|6|0|0|2
5|6|0|4|2
5|6|0|4|2
5|6|0|5|3
5|6|0|9|9
5|6|0|2|7
5|6|0|8|1
5|6|0|6|3
5|6|0|3|1
5|6|0|5|1
5|6|0|0|3
runtime (debug build, this branch): total: 9 ms (this includes parsing/coloring of cli app)
runtime (debug build, main branch): total: 71 ms (this includes parsing/coloring of cli app)
```
Closes #1288
2025-04-11 09:36:25 +03:00
Pekka Enberg
d67e1b604b
Merge 'Added 'likelihood' scalar function' from Sachin Kumar Singh
...
The `likelihood(X,Y)` function returns argument X unchanged. The value Y
in likelihood(X,Y) must be a floating point constant between 0.0 and
1.0, inclusive.
```
sqlite> explain SELECT likelihood(42, 0.0);
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 6 0 0 Start at 6
1 Once 0 3 0 0
2 Integer 42 2 0 0 r[2]=42
3 Copy 2 1 0 0 r[1]=r[2]
4 ResultRow 1 1 0 0 output=r[1]
5 Halt 0 0 0 0
6 Goto 0 1 0 0
```
```
limbo> explain SELECT likelihood(42, 0.0);
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 4 0 0 Start at 4
1 Copy 2 1 0 0 r[1]=r[2]
2 ResultRow 1 1 0 0 output=r[1]
3 Halt 0 0 0 0
4 Integer 42 2 0 0 r[2]=42
5 Goto 0 1 0 0
```
Closes #1303
2025-04-11 09:34:36 +03:00
Sachin Singh
01fa02364d
correctly handle edge cases
2025-04-11 08:34:29 +05:30
Sachin Singh
482e93bfd0
feat: add likelihood scalar function
2025-04-11 05:54:23 +05:30
Sachin Singh
b7acfa490c
feat: add timediff data and time function
2025-04-11 04:30:57 +05:30
Jussi Saurio
4daad0a858
Fix bug: accidentally skipped index selection for other tables except first found
2025-04-10 18:57:14 +03:00
Jussi Saurio
457bded14d
optimizer: refactor optimizer to support multicolumn index scans
2025-04-10 15:53:02 +03:00
PThorpe92
f223e66c82
Remove unused mut and fix merge conflict issues
2025-04-09 11:15:04 -04:00
PThorpe92
62d1447cd6
Adapt query plan to handle vatbs for updates
2025-04-09 11:15:02 -04:00
PThorpe92
0ffecb3021
Add comments to document update on vtabs
2025-04-09 11:06:41 -04:00
PThorpe92
b685086cad
Support UPDATE for virtual tables
2025-04-09 11:06:41 -04:00
Pekka Enberg
a4d9f70ef8
Merge 'Strict table support' from Ihor Andrianov
...
Closes #884
Support for
```CREATE TABLE test(id INTEGER) STRICT;```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1268
2025-04-09 16:45:56 +03:00
Jussi Saurio
3124fca5b7
Dereference instead of explicit clone
2025-04-09 10:14:29 +03:00
Jussi Saurio
024c63f808
optimizer: remove ORDER BY if index can be used to satisfy the order
2025-04-09 10:14:29 +03:00
Jussi Saurio
a706b7160a
planner: support index backwards seeks and iteration
2025-04-09 10:14:29 +03:00
Jussi Saurio
ed929d2a0a
Merge 'Properly handle insertion of indexed columns' from Preston Thorpe
...
```console
limbo> create index p on products(price);
limbo> explain insert into products (name,price) values ('huh',32);
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 17 0 0 Start at 17
1 OpenWriteAsync 0 273 0 0
2 OpenWriteAwait 0 0 0 0
3 String8 0 3 0 huh 0 r[3]='huh'
4 Integer 32 4 0 0 r[4]=32
5 OpenWriteAsync 1 297 0 0
6 OpenWriteAwait 0 0 0 0
7 NewRowId 0 1 0 0
8 MakeRecord 2 3 5 0 r[5]=mkrec(r[2..4])
9 InsertAsync 0 5 1 0
10 InsertAwait 0 0 0 0
11 Copy 4 6 0 0 r[6]=r[4]
12 Copy 1 7 0 0 r[7]=r[1]
13 MakeRecord 6 2 8 0 r[8]=mkrec(r[6..7])
14 IdxInsertAsync 1 8 6 0 key=r[8]
15 IdxInsertAwait 1 0 0 0
16 Halt 0 0 0 0
17 Transaction 0 1 0 0 write=true
18 Null 0 2 0 0 r[2]=NULL
19 Goto 0 1 0 0
```
When an insert happens, we need to be sure to insert into the index
btree as well.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1265
2025-04-09 10:13:59 +03:00
PThorpe92
c15035caf8
Add module and vtab to schema after table is reopened with proper ext
2025-04-08 20:10:48 -04:00
PThorpe92
224f913ae7
Handle composite key indexes on insert
2025-04-08 08:55:14 -04:00
PThorpe92
878c987026
Remove is_null check from create index translation
2025-04-08 08:55:14 -04:00
PThorpe92
1f29307fe8
Support proper index handling when doing insertions
2025-04-08 08:55:14 -04:00
Ihor Andrianov
7c15465118
add TypeCheck insn to update
2025-04-07 20:02:14 +03:00
Ihor Andrianov
4a08b98bab
implemented strict table
2025-04-07 20:01:39 +03:00
Duncan Lutz
aa7c64cb19
feat: added likely scalar function
2025-04-06 23:14:30 -06:00
Jussi Saurio
c19e4fc69c
Merge 'Aggregation without group by produces incorrect results for scalars' from Ihor Andrianov
...
Closes #954
Before:
<img width="669" alt="Знімок екрана 2025-03-27 о 21 49 19"
src="https://github.com/user-
attachments/assets/d005e690-7dab-41e5-bc03-b574cade3965" />
After:
<img width="676" alt="Знімок екрана 2025-03-27 о 21 49 44"
src="https://github.com/user-
attachments/assets/1f4eb6bf-a238-496e-9fa4-32382799ef86" />
SQLite:
<img width="656" alt="Знімок екрана 2025-03-27 о 21 50 04"
src="https://github.com/user-
attachments/assets/3eca184e-6ea5-47c1-824f-51d11256a7af" />
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com >
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1198
2025-04-06 20:00:30 +03:00
Preston Thorpe
62e0a137dd
Merge branch 'main' into update_offset
2025-04-06 12:34:23 -04:00
Pekka Enberg
2d3fd01f91
Merge 'Support Create Index' from Preston Thorpe
...
Closes #1193
```console
│limbo> explain create index idxp on products(price);
│addr opcode p1 p2 p3 p4 p5 comment
│---- ----------------- ---- ---- ---- ------------- -- -------
│0 Init 0 39 0 0 Start at 39
│1 CreateBtree 0 1 2 0 r[1]=root iDb=0 flags=2
│2 OpenWriteAsync 0 1 0 0
│3 OpenWriteAwait 0 0 0 0
│4 NewRowId 0 2 0 0
│5 String8 0 3 0 index 0 r[3]='index'
│6 String8 0 4 0 idxp 0 r[4]='idxp'
│7 String8 0 5 0 products 0 r[5]='products'
│8 Copy 1 6 1 0 r[6]=r[1]
│9 String8 0 7 0 CREATE INDEX idxp ON products (price) 0 r[7]='CREATE INDEX idxp ON products (price)'
│10 MakeRecord 3 5 8 0 r[8]=mkrec(r[3..7])
│11 InsertAsync 0 8 2 0
│12 InsertAwait 0 0 0 0
│13 SorterOpen 3 1 0 k(1,B) 0 cursor=3
│14 OpenPseudo 4 9 2 0 2 columns in r[9]
│15 OpenReadAsync 2 273 0 0 table=products, root=273
│16 OpenReadAwait 0 0 0 0
│17 RewindAsync 2 0 0 0
│18 RewindAwait 2 25 0 0 Rewind table products
│19 Column 2 2 10 0 r[10]=products.price
│20 RowId 2 11 0 0 r[11]=products.rowid
│21 MakeRecord 10 2 12 0 r[12]=mkrec(r[10..11])
│22 SorterInsert 3 12 0 0 0 key=r[12]
│23 NextAsync 2 0 0 0
│24 NextAwait 2 19 0 0
│25 OpenWriteAsync 1 1 0 0
│26 OpenWriteAwait 0 0 0 0
│27 SorterSort 3 33 0 0
│28 SorterData 3 13 4 0 r[13]=data
│29 SeekEnd 1 0 0 0
│30 IdxInsertAsync 1 13 0 0 key=r[13]
│31 IdxInsertAwait 1 0 0 0
│32 SorterNext 3 28 0 0
│33 Close 3 0 0 0
│34 Close 2 0 0 0
│35 Close 1 0 0 0
│36 ParseSchema 0 0 0 name = 'idxp' AND type = 'index' 0 name = 'idxp' AND type = 'index'
│37 Close 0 0 0 0
│38 Halt 0 0 0 0
│39 Transaction 0 1 0 0 write=true
│40 Goto 0 1 0 0
```
This will create the initial index btree and insert whatever relevant
records that need to be inserted, it doesn't handle the case of
inserting new index keys when normal records are created afterwards.
That will prob be added in next PR to keep this one concise.
Limbo will properly use the index in a subsequent query:
Creating a unique index on a column that has 2 existing identical rows:
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com >
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1199
2025-04-06 19:32:36 +03:00
PThorpe92
c9edf71fb8
Support insert default values syntax
2025-04-05 21:27:21 -04:00
PThorpe92
007fbe8cc7
Fix unique index issue and prealloc in sql string for schema
2025-04-05 11:19:09 -04:00
PThorpe92
2c3fd509fe
Remove unused imports and consolidate ordering comparison
2025-04-05 11:19:09 -04:00
PThorpe92
e020ba3dfe
Add enum for interpreting a value as a register or literal for insns
2025-04-05 11:19:07 -04:00
PThorpe92
b0016a0ee2
Support create index with SeekEnd and IdxCreate opcode functionality
2025-04-05 11:15:36 -04:00
PThorpe92
4741544dfd
Add query translation for create index
2025-04-05 11:06:18 -04:00
Ihor Andrianov
0c9464e3fc
reduce vec allocations, add comments for magic ifs
2025-04-05 15:15:10 +03:00
PThorpe92
e3985b6994
Remove unused mut ref from emit_update_instructions for tx context
2025-04-04 12:51:37 -04:00
PThorpe92
13e084351d
Change parse_limit function to accept reference value to ast::Limit
2025-04-04 12:38:18 -04:00
PThorpe92
f6a64a7b15
Support OFFSET clause for LIMIT in UPDATE queries
2025-04-04 12:35:30 -04:00
Pekka Enberg
c3eaf47180
Merge 'Add support for default values in INSERT statements' from Diego Reis
...
While working on #494 I noticed that default values defined in schemas
weren't being applied.
Before:
Now:
Closes #1249
2025-04-04 08:59:44 +03:00
Diego Reis
43daba9942
core/translate: Add support for default values in INSERT statements
2025-04-04 01:32:13 -03:00
PThorpe92
ae2be30204
Move init label to proper place in create vtab translation
2025-04-03 20:22:14 -04:00
Ihor Andrianov
d4b8fa17f8
fix tests
2025-04-03 22:28:14 +03:00
Ihor Andrianov
34a132fcd3
fix output when group by is not part of resulting set
2025-04-03 22:28:13 +03:00
Ihor Andrianov
91ceab1626
improve naming and add comments for context
2025-04-03 22:28:13 +03:00
Ihor Andrianov
816cbacc9c
some smartie optimizations
2025-04-03 22:28:12 +03:00
Ihor Andrianov
2bcdd4e404
non group by cols are displayed in group by agg statements
2025-04-03 22:28:12 +03:00
Ihor Andrianov
4fd1dcdc73
small refine
2025-04-03 22:28:11 +03:00
