Pere Diaz Bou
6b7575bf3f
fix tree traversal assumptions on traversal
2025-04-09 15:04:45 +02:00
Pere Diaz Bou
f2d9e1e8f5
fix divider cell in index
2025-04-09 15:04:45 +02:00
Pere Diaz Bou
12899034c9
make insert idx re-entrant
2025-04-09 15:04:45 +02:00
Pere Diaz Bou
d9453f6e06
fix cell_get_raw_region length calculation
2025-04-09 15:04:45 +02:00
Pere Diaz Bou
0f59fc7e36
Merge 'Decrease page count on balancing fixes' from Pere Diaz Bou
...
* Comment how decrease of pages happen while balancing
* Free pages no longer used after balancing finished.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1278
2025-04-09 15:04:25 +02:00
Pere Diaz Bou
f1df09ffd9
free no longer used pages after balance
2025-04-09 11:12:39 +02:00
Jussi Saurio
aa6e2d853a
Merge 'Support backwards index scan and seeks + utilize indexes in removing ORDER BY' from Jussi Saurio
...
## Main stuff
- Support iterating an index backwards
- Support scanning an index (instead of seeking with a condition)
- Support backwards index seeks
- Support backwards rowid seeks
- Fix existing backwards iteration logic for table btrees
- Remove ORDER BY entirely if any index satisfies the ordering
- Add fuzz tests for rowid seeks, 1 and 2 column index seeks
## Bytecode examples (note the lack of order by sorting):
one column index order by, forwards:
```sql
limbo> explain select first_name from users order by age;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 13 0 0 Start at 13
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 RewindAsync 1 0 0 0
6 RewindAwait 1 12 0 0 Rewind table age_idx
7 DeferredSeek 1 0 0 0
8 Column 0 1 1 0 r[1]=users.first_name
9 ResultRow 1 1 0 0 output=r[1]
10 NextAsync 1 0 0 0
11 NextAwait 1 7 0 0
12 Halt 0 0 0 0
13 Transaction 0 0 0 0 write=false
14 Goto 0 1 0 0
```
one column index order by, backwards:
```sql
limbo> explain select first_name from users order by age desc;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 13 0 0 Start at 13
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 LastAsync 1 0 0 0
6 LastAwait 1 0 0 0
7 DeferredSeek 1 0 0 0
8 Column 0 1 1 0 r[1]=users.first_name
9 ResultRow 1 1 0 0 output=r[1]
10 PrevAsync 1 0 0 0
11 PrevAwait 1 0 0 0
12 Halt 0 0 0 0
13 Transaction 0 0 0 0 write=false
14 Goto 0 1 0 0
```
rowid seek, backwards:
```sql
limbo> explain select * from users where id < 100 order by id desc;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 19 0 0 Start at 19
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 Integer 100 11 0 0 r[11]=100
4 SeekLT 0 18 11 0
5 RowId 0 1 0 0 r[1]=users.rowid
6 Column 0 1 2 0 r[2]=users.first_name
7 Column 0 2 3 0 r[3]=users.last_name
8 Column 0 3 4 0 r[4]=users.email
9 Column 0 4 5 0 r[5]=users.phone_number
10 Column 0 5 6 0 r[6]=users.address
11 Column 0 6 7 0 r[7]=users.city
12 Column 0 7 8 0 r[8]=users.state
13 Column 0 8 9 0 r[9]=users.zipcode
14 Column 0 9 10 0 r[10]=users.age
15 ResultRow 1 10 0 0 output=r[1..10]
16 PrevAsync 0 0 0 0
17 PrevAwait 0 0 0 0
18 Halt 0 0 0 0
19 Transaction 0 0 0 0 write=false
20 Goto 0 1 0 0
```
two column order by, setup:
```sql
cargo run dualpk.db
Limbo v0.0.18-pre.3
Enter ".help" for usage hints.
limbo> .schema
CREATE TABLE a(b,c,d,e, primary key (d,c));
```
two column order by, forwards:
```sql
limbo> explain select * from a order by d,c;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 16 0 0 Start at 16
1 OpenReadAsync 0 2 0 0 table=a, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=sqlite_autoindex_a_1, root=3
4 OpenReadAwait 0 0 0 0
5 RewindAsync 1 0 0 0
6 RewindAwait 1 15 0 0 Rewind table sqlite_autoindex_a_1
7 DeferredSeek 1 0 0 0
8 Column 0 0 1 0 r[1]=a.b
9 Column 0 1 2 0 r[2]=a.c
10 Column 0 2 3 0 r[3]=a.d
11 Column 0 3 4 0 r[4]=a.e
12 ResultRow 1 4 0 0 output=r[1..4]
13 NextAsync 1 0 0 0
14 NextAwait 1 7 0 0
15 Halt 0 0 0 0
16 Transaction 0 0 0 0 write=false
17 Goto 0 1 0 0
```
two column order by, forwards with index seek:
```sql
limbo> explain select * from a where d > 100 order by d,c;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 16 0 0 Start at 16
1 OpenReadAsync 0 2 0 0 table=a, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=sqlite_autoindex_a_1, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 100 5 0 0 r[5]=100
6 SeekGT 1 15 5 0
7 DeferredSeek 1 0 0 0
8 Column 0 0 1 0 r[1]=a.b
9 Column 0 1 2 0 r[2]=a.c
10 Column 0 2 3 0 r[3]=a.d
11 Column 0 3 4 0 r[4]=a.e
12 ResultRow 1 4 0 0 output=r[1..4]
13 NextAsync 1 0 0 0
14 NextAwait 1 7 0 0
15 Halt 0 0 0 0
16 Transaction 0 0 0 0 write=false
17 Goto 0 1 0 0
```
two column order by, forwards with index scan and termination condition:
```sql
limbo> explain select * from a where d < 100 order by d,c;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 18 0 0 Start at 18
1 OpenReadAsync 0 2 0 0 table=a, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=sqlite_autoindex_a_1, root=3
4 OpenReadAwait 0 0 0 0
5 Null 0 5 0 0 r[5]=NULL
6 SeekGT 1 17 5 0
7 Integer 100 6 0 0 r[6]=100
8 IdxGE 1 17 6 0
9 DeferredSeek 1 0 0 0
10 Column 0 0 1 0 r[1]=a.b
11 Column 0 1 2 0 r[2]=a.c
12 Column 0 2 3 0 r[3]=a.d
13 Column 0 3 4 0 r[4]=a.e
14 ResultRow 1 4 0 0 output=r[1..4]
15 NextAsync 1 0 0 0
16 NextAwait 1 7 0 0
17 Halt 0 0 0 0
18 Transaction 0 0 0 0 write=false
19 Goto 0 1 0 0
```
two column order by, backwards:
```sql
limbo> explain select * from a order by d desc,c desc;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 16 0 0 Start at 16
1 OpenReadAsync 0 2 0 0 table=a, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=sqlite_autoindex_a_1, root=3
4 OpenReadAwait 0 0 0 0
5 LastAsync 1 0 0 0
6 LastAwait 1 0 0 0
7 DeferredSeek 1 0 0 0
8 Column 0 0 1 0 r[1]=a.b
9 Column 0 1 2 0 r[2]=a.c
10 Column 0 2 3 0 r[3]=a.d
11 Column 0 3 4 0 r[4]=a.e
12 ResultRow 1 4 0 0 output=r[1..4]
13 PrevAsync 1 0 0 0
14 PrevAwait 1 0 0 0
15 Halt 0 0 0 0
16 Transaction 0 0 0 0 write=false
17 Goto 0 1 0 0
```
two column order by, backwards with index seek:
```sql
limbo> explain select * from a where d < 100 order by d desc,c desc;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 16 0 0 Start at 16
1 OpenReadAsync 0 2 0 0 table=a, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=sqlite_autoindex_a_1, root=3
4 OpenReadAwait 0 0 0 0
5 Integer 100 5 0 0 r[5]=100
6 SeekLT 1 15 5 0
7 DeferredSeek 1 0 0 0
8 Column 0 0 1 0 r[1]=a.b
9 Column 0 1 2 0 r[2]=a.c
10 Column 0 2 3 0 r[3]=a.d
11 Column 0 3 4 0 r[4]=a.e
12 ResultRow 1 4 0 0 output=r[1..4]
13 PrevAsync 1 0 0 0
14 PrevAwait 1 0 0 0
15 Halt 0 0 0 0
16 Transaction 0 0 0 0 write=false
17 Goto 0 1 0 0
```
two column order by, backwards with index scan and termination
condition:
```sql
limbo> explain select * from a where d > 100 order by d desc,c desc;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 18 0 0 Start at 18
1 OpenReadAsync 0 2 0 0 table=a, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=sqlite_autoindex_a_1, root=3
4 OpenReadAwait 0 0 0 0
5 LastAsync 1 0 0 0
6 LastAwait 1 0 0 0
7 Integer 100 6 0 0 r[6]=100
8 IdxLE 1 17 6 0
9 DeferredSeek 1 0 0 0
10 Column 0 0 1 0 r[1]=a.b
11 Column 0 1 2 0 r[2]=a.c
12 Column 0 2 3 0 r[3]=a.d
13 Column 0 3 4 0 r[4]=a.e
14 ResultRow 1 4 0 0 output=r[1..4]
15 PrevAsync 1 0 0 0
16 PrevAwait 1 0 0 0
17 Halt 0 0 0 0
18 Transaction 0 0 0 0 write=false
19 Goto 0 1 0 0
```
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes #1209
2025-04-09 12:03:14 +03:00
Pere Diaz Bou
edc3a420fb
comment how page count is decreased while balancing
2025-04-09 11:02:49 +02:00
Jussi Saurio
0888c71ba0
use seek() instead of do_seek() to set iteration state
2025-04-09 10:26:02 +03:00
Jussi Saurio
7435d005db
Merge ' allow insertion of multiple overflow cells ' from Pere Diaz Bou
...
With big cells it is easy to see multiple overflow cells happen while
balancing. Previously we disallowed insertion with > 1 overflow cells,
let's fix this by allowing it as we should be safe now.
This pr also adds some more comments to the balancing algorithm.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1270
2025-04-09 10:17:36 +03:00
Jussi Saurio
5e3a37a192
Try to name iteration direction sensitive method better
2025-04-09 10:14:29 +03:00
Jussi Saurio
3124fca5b7
Dereference instead of explicit clone
2025-04-09 10:14:29 +03:00
Jussi Saurio
0bb87b060a
Fix existing table btree backwards iteration logic
2025-04-09 10:14:29 +03:00
Jussi Saurio
d9bae633c0
Add rowid_seek_fuzz() test
2025-04-09 10:14:29 +03:00
Jussi Saurio
f5220d281d
Fix off-by-one logic in btree table traversal
2025-04-09 10:14:29 +03:00
Jussi Saurio
fa295af635
Fix insert fuzz test by bypassing internal invariant
2025-04-09 10:14:29 +03:00
Jussi Saurio
024c63f808
optimizer: remove ORDER BY if index can be used to satisfy the order
2025-04-09 10:14:29 +03:00
Jussi Saurio
a706b7160a
planner: support index backwards seeks and iteration
2025-04-09 10:14:29 +03:00
Jussi Saurio
c9190236f0
btree: support backwards index seeks and iteration
2025-04-09 10:14:29 +03:00
Jussi Saurio
3e42a62cd0
Add SeekLE/SeekLT operations to VDBE
2025-04-09 10:14:29 +03:00
Jussi Saurio
431ef2fa6a
Add TCL/differential fuzz tests for verifying index scan behavior
2025-04-09 10:14:29 +03:00
Jussi Saurio
ed929d2a0a
Merge 'Properly handle insertion of indexed columns' from Preston Thorpe
...
```console
limbo> create index p on products(price);
limbo> explain insert into products (name,price) values ('huh',32);
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 17 0 0 Start at 17
1 OpenWriteAsync 0 273 0 0
2 OpenWriteAwait 0 0 0 0
3 String8 0 3 0 huh 0 r[3]='huh'
4 Integer 32 4 0 0 r[4]=32
5 OpenWriteAsync 1 297 0 0
6 OpenWriteAwait 0 0 0 0
7 NewRowId 0 1 0 0
8 MakeRecord 2 3 5 0 r[5]=mkrec(r[2..4])
9 InsertAsync 0 5 1 0
10 InsertAwait 0 0 0 0
11 Copy 4 6 0 0 r[6]=r[4]
12 Copy 1 7 0 0 r[7]=r[1]
13 MakeRecord 6 2 8 0 r[8]=mkrec(r[6..7])
14 IdxInsertAsync 1 8 6 0 key=r[8]
15 IdxInsertAwait 1 0 0 0
16 Halt 0 0 0 0
17 Transaction 0 1 0 0 write=true
18 Null 0 2 0 0 r[2]=NULL
19 Goto 0 1 0 0
```
When an insert happens, we need to be sure to insert into the index
btree as well.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1265
2025-04-09 10:13:59 +03:00
Jussi Saurio
08b352cd81
Merge 'VTabs: Proper handling of re-opened db files without the relevant extensions loaded' from Preston Thorpe
...
closes #1239
warns the user if a .db file is opened that contains a schema entry of a
vtab created with a module that is no longer loaded.
If the module is loaded for that connection, the user can properly query
the table.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1244
2025-04-09 09:36:26 +03:00
PThorpe92
9b1e60a29c
Fix typo in ext library lock err message
2025-04-08 20:10:50 -04:00
PThorpe92
41ac91f14f
Add tests for parsing vtab creation sql in ParseSchema
2025-04-08 20:10:49 -04:00
PThorpe92
3a7f1e4056
Add comments explaining flow of reloading vtabs from schema tbl
2025-04-08 20:10:49 -04:00
PThorpe92
6b5ec1f07b
Remove mut borrow from sym table in parse schema fn
2025-04-08 20:10:49 -04:00
PThorpe92
a0f71e27be
Fix cli tests
2025-04-08 20:10:49 -04:00
PThorpe92
c15035caf8
Add module and vtab to schema after table is reopened with proper ext
2025-04-08 20:10:48 -04:00
PThorpe92
4b9b6c969b
Parse schema rows after extensions are loaded
2025-04-08 20:10:47 -04:00
PThorpe92
3ad7d194cb
Prevent panic on loading non-existent vtab module
2025-04-08 20:09:27 -04:00
Pere Diaz Bou
ce7e0188f6
bring back i64 page sizes while balancing
2025-04-08 17:57:39 +02:00
Pere Diaz Bou
cf62099bf5
allow insertion of multiple overflow cells
2025-04-08 16:43:15 +02:00
Pere Diaz Bou
029da5c81c
Improve readability of balance_non_root with comments and validation extraction
2025-04-08 16:43:15 +02:00
Pere Diaz Bou
e368cd1499
Merge 'Account divider cell in size while distributing cells' from Pere Diaz Bou
...
~~Moving cells from right to left involves moving cell that are from any
page in the right, from left to right, this means we update the size of
page of adjacent one only but not the last one. This is kinda weird but
it works. I might change it if it feels to weird.~~
Update: my previous analysis was obviously incorrect. Underflowing is
undesired behaviour and accounting for divider cell avoided it.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1269
2025-04-08 16:42:16 +02:00
Pekka Enberg
0ec12b0192
Merge 'Format infinite float as "Inf"/"-Inf"' from jachewz
...
## Issue
Numeric values outside the range of what f64 can represent are formatted
to strings as "Inf" or "-Inf" in sqlite, but they are "inf" and "-inf"
in limbo.
```
sqlite> SELECT 1.7976931348623157E+309;
Inf
```
```
limbo> SELECT 1.7976931348623157E+309;
┌─────────────────────────┐
│ 1.7976931348623157E+309 │
├─────────────────────────┤
│ inf │
└─────────────────────────┘
```
closes #1248
Closes #1272
2025-04-08 16:52:36 +03:00
jachewz
12ae07874e
fmt inf float str as "Inf"/"-Inf"
2025-04-08 23:33:34 +10:00
PThorpe92
224f913ae7
Handle composite key indexes on insert
2025-04-08 08:55:14 -04:00
PThorpe92
878c987026
Remove is_null check from create index translation
2025-04-08 08:55:14 -04:00
PThorpe92
1f29307fe8
Support proper index handling when doing insertions
2025-04-08 08:55:14 -04:00
Pere Diaz Bou
fded6ccaf3
rever iterations fuzz test
2025-04-08 14:09:17 +02:00
Pere Diaz Bou
c0c66bf8af
remove wrong comment
2025-04-08 14:06:48 +02:00
Pere Diaz Bou
8c4003908f
bring back usize, it shouldn't underflow
2025-04-08 14:05:30 +02:00
Pere Diaz Bou
40f8bbe132
clippy
2025-04-08 11:31:38 +02:00
Pere Diaz Bou
8e88b0cd14
new_page_sizes as Vec<i64>
2025-04-08 11:31:38 +02:00
Pere Diaz Bou
3950ab1e52
account for divider cell size in page size
2025-04-08 11:31:38 +02:00
Pere Diaz Bou
f7de575873
Merge 'update sqlite download version to 2025 + remove www.' from Pere Diaz Bou
...
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #1271
2025-04-08 11:30:53 +02:00
Pere Diaz Bou
b83b51e973
remove www. prefix
2025-04-08 11:16:32 +02:00
Pere Diaz Bou
03f531417c
update sqlite download version to 2025
2025-04-08 11:12:20 +02:00
Pekka Enberg
33afc7cef3
Merge 'Improve validation of btree balancing' from Pere Diaz Bou
...
This pr sprinkles debug compile check to ensure things stay in a
reliable state.
Closes #1261
2025-04-07 19:35:00 +03:00
