Pere Diaz Bou
cef78d54de
fix generic
2024-11-06 17:53:14 +01:00
Pere Diaz Bou
df45f17486
fix fmt
2024-11-06 17:47:41 +01:00
Pere Diaz Bou
eb8c462c5f
fix io submission on cacheflush
2024-11-06 16:25:42 +00:00
Pere Diaz Bou
8eb3c89227
wasm,sim fixes
2024-11-05 15:41:30 +01:00
Pere Diaz Bou
a85d599c65
state machine cacheflush
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
da7717edfb
fix generic io match
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
5207e49399
remove extra borrow mut
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
f43e53a815
fix wasm and simulator wal impl
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
0f4270b48f
rebase submit entry
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
70a4ccd8bb
fix linux completion match
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
fc65c5096d
cacheflush state machine
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
129cc1cd6d
fix open_file generic
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
f66e3925f3
fix imports
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
c2453a084d
test wal checkpoint
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
f009eb35c6
suspendable checkpoint
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
c0e51c4ca6
wip wal
2024-11-05 15:29:53 +01:00
Pekka Enberg
0a313c6486
Merge 'Ignore .vscode directory' from Lauri Virtanen
...
Closes #370
2024-10-14 08:04:25 +03:00
Lauri Virtanen
b90a0b3754
Ignore .vscode directory
2024-10-14 00:00:33 +03:00
jussisaurio
7b7dac8ec5
Merge 'clippy fix' from Jussi Saurio
...
Closes #369
2024-10-13 12:21:30 +03:00
jussisaurio
daf5863932
manual fixes based on clippy suggestions
2024-10-13 12:19:04 +03:00
jussisaurio
f634e7f7a3
clippy fix
2024-10-13 12:08:54 +03:00
jussisaurio
bfdd4bc513
Merge 'core/translate: (refactor) use btreetablereference struct instead of tuple' from Jussi Saurio
...
`table` (a `Rc<BTreeTable>`) and `table_identifier` (a `String`) pretty
much always go together in the current implementation, so makes sense to
colocate them as a struct instead of passing around a tuple everywhere
Closes #368
2024-10-13 12:05:19 +03:00
jussisaurio
b3c57d5691
core/translate: (refactor) use btreetablereference struct instead of tuple
2024-10-13 11:25:33 +03:00
Pekka Enberg
d8172ef20b
Update CHANGELOG.md
2024-10-13 10:13:06 +03:00
Pekka Enberg
bf2e254a21
Merge 'core/translate/optimizer: eliminate unnecessary ORDER BY if result set is already ordered' from Jussi Saurio
...
Built on top of, and currently targets, #350 , _not_ main
Closes #365
Examples:
```
limbo> explain select u.first_name, p.name from users u join products p on u.id = p.id order by u.id;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 15 0 0 Start at 15
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 RewindAsync 0 0 0 0
6 RewindAwait 0 14 0 0 Rewind table u
7 RowId 0 1 0 0 r[1]=u.rowid
8 SeekRowid 1 1 12 0 if (r[1]!=p.rowid) goto 12
9 Column 0 1 2 0 r[2]=u.first_name
10 Column 1 1 3 0 r[3]=p.name
11 ResultRow 2 2 0 0 output=r[2..3]
12 NextAsync 0 0 0 0
13 NextAwait 0 7 0 0
14 Halt 0 0 0 0
15 Transaction 0 0 0 0
16 Goto 0 1 0 0
```
```
limbo> explain select * from users where age > 80 order by age limit 5;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 23 0 0 Start at 23
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 Integer 80 1 0 0 r[1]=80
6 SeekGT 1 22 1 0
7 DeferredSeek 1 0 0 0
8 RowId 0 2 0 0 r[2]=users.rowid
9 Column 0 1 3 0 r[3]=users.first_name
10 Column 0 2 4 0 r[4]=users.last_name
11 Column 0 3 5 0 r[5]=users.email
12 Column 0 4 6 0 r[6]=users.phone_number
13 Column 0 5 7 0 r[7]=users.address
14 Column 0 6 8 0 r[8]=users.city
15 Column 0 7 9 0 r[9]=users.state
16 Column 0 8 10 0 r[10]=users.zipcode
17 Column 0 9 11 0 r[11]=users.age
18 ResultRow 2 10 0 0 output=r[2..11]
19 DecrJumpZero 12 22 0 0 if (--r[12]==0) goto 22
20 NextAsync 1 0 0 0
21 NextAwait 1 7 0 0
22 Halt 0 0 0 0
23 Transaction 0 0 0 0
24 Integer 5 12 0 0 r[12]=5
25 Goto 0 1 0 0
```
Closes #366
2024-10-13 10:12:28 +03:00
Pekka Enberg
fdf3b0d16b
Merge '(core): Primary key index scans and single-column secondary index scans' from Jussi Saurio
...
This PR adds an index on `users.age` to `testing.db`, and support for
indexed lookups. Only single-column ascending indexes are currently
supported.
This PR also gets rid of `Operator::Seekrowid` in favor of
`Operator::Search` which handles all non-full-table-scan searches: 1.
integer primary key (rowid) point queries 2. integer primary key index
scans, and 3. secondary index scans.
examples:
```
limbo> select first_name, age from users where age > 90 limit 10;
Miranda|90
Sarah|90
Justin|90
Justin|90
John|90
Jeremy|90
Stephanie|90
Joshua|90
Jenny|90
Jennifer|90
limbo> explain query plan select first_name, age from users where age > 90 limit 10;
QUERY PLAN
`--TAKE 10
`--PROJECT first_name, age
| `--SEARCH users USING INDEX age_idx
limbo> explain select first_name, age from users where age > 90 limit 10;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 15 0 0 Start at 15
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 Integer 90 1 0 0 r[1]=90
6 SeekGT 1 14 1 0
7 DeferredSeek 1 0 0 0
8 Column 0 1 2 0 r[2]=users.first_name
9 Column 0 9 3 0 r[3]=users.age
10 ResultRow 2 2 0 0 output=r[2..3]
11 DecrJumpZero 4 14 0 0 if (--r[4]==0) goto 14
12 NextAsync 1 0 0 0
13 NextAwait 1 7 0 0
14 Halt 0 0 0 0
15 Transaction 0 0 0 0
16 Integer 10 4 0 0 r[4]=10
17 Goto 0 1 0 0
```
Sqlite version:
```
sqlite> explain select first_name, age from users where age > 90 limit 10;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 13 0 0 Start at 13
1 Integer 10 1 0 0 r[1]=10; LIMIT counter
2 OpenRead 0 2 0 10 0 root=2 iDb=0; users
3 OpenRead 1 274 0 k(2,,) 0 root=274 iDb=0; age_idx
4 Integer 90 2 0 0 r[2]=90
5 SeekGT 1 12 2 1 0 key=r[2]
6 DeferredSeek 1 0 0 0 Move 0 to 1.rowid if needed
7 Column 0 1 3 0 r[3]= cursor 0 column 1
8 Column 1 0 4 0 r[4]= cursor 1 column 0
9 ResultRow 3 2 0 0 output=r[3..4]
10 DecrJumpZero 1 12 0 0 if (--r[1])==0 goto 12
11 Next 1 6 0 0
12 Halt 0 0 0 0
13 Transaction 0 0 3 0 1 usesStmtJournal=0
14 Goto 0 1 0 0
```
---
´Seek` instructions are also now supported for primary key rowid
searches:
```
limbo> select id, first_name from users where id > 9995;
9996|Donald
9997|Ruth
9998|Dorothy
9999|Gina
10000|Nicole
limbo> explain query plan select id, first_name from users where id > 9995;
QUERY PLAN
`--PROJECT id, first_name
`--SEARCH users USING INTEGER PRIMARY KEY (rowid=?)
limbo> explain select id, first_name from users where id > 9995;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 11 0 0 Start at 11
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 Integer 9995 1 0 0 r[1]=9995
4 SeekGT 0 10 1 0
5 RowId 0 2 0 0 r[2]=users.rowid
6 Column 0 1 3 0 r[3]=users.first_name
7 ResultRow 2 2 0 0 output=r[2..3]
8 NextAsync 0 0 0 0
9 NextAwait 0 5 0 0
10 Halt 0 0 0 0
11 Transaction 0 0 0 0
12 Goto 0 1 0 0
```
sqlite:
```
sqlite> explain select id, first_name from users where id > 9995;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 8 0 0 Start at 8
1 OpenRead 0 2 0 2 0 root=2 iDb=0; users
2 SeekGT 0 7 1 0 key=r[1]; pk
3 Rowid 0 2 0 0 r[2]=users.rowid
4 Column 0 1 3 0 r[3]= cursor 0 column 1
5 ResultRow 2 2 0 0 output=r[2..3]
6 Next 0 3 0 0
7 Halt 0 0 0 0
8 Transaction 0 0 3 0 1 usesStmtJournal=0
9 Integer 9995 1 0 0 r[1]=9995
10 Goto 0 1 0 0
```
---
More complex example with a join that uses both a rowid lookup and a
secondary index scan:
```
limbo> explain query plan select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
QUERY PLAN
`--PROJECT u.first_name, p.name
`--JOIN
| |--SEARCH u USING INDEX age_idx
| `--SEARCH p USING INTEGER PRIMARY KEY (rowid=?)
limbo> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 18 0 0 Start at 18
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 OpenReadAsync 2 3 0 0 table=p, root=3
6 OpenReadAwait 0 0 0 0
7 Integer 70 1 0 0 r[1]=70
8 SeekGT 1 17 1 0
9 DeferredSeek 1 0 0 0
10 RowId 0 2 0 0 r[2]=u.rowid
11 SeekRowid 2 2 15 0 if (r[2]!=p.rowid) goto 15
12 Column 0 1 3 0 r[3]=u.first_name
13 Column 2 1 4 0 r[4]=p.name
14 ResultRow 3 2 0 0 output=r[3..4]
15 NextAsync 1 0 0 0
16 NextAwait 1 9 0 0
17 Halt 0 0 0 0
18 Transaction 0 0 0 0
19 Goto 0 1 0 0
```
sqlite:
```
sqlite> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 14 0 0 Start at 14
1 OpenRead 0 2 0 10 0 root=2 iDb=0; users
2 OpenRead 2 274 0 k(2,,) 0 root=274 iDb=0; age_idx
3 OpenRead 1 3 0 2 0 root=3 iDb=0; products
4 Integer 70 1 0 0 r[1]=70
5 SeekGT 2 13 1 1 0 key=r[1]
6 DeferredSeek 2 0 0 0 Move 0 to 2.rowid if needed
7 IdxRowid 2 2 0 0 r[2]=rowid; users.rowid
8 SeekRowid 1 12 2 0 intkey=r[2]
9 Column 0 1 3 0 r[3]= cursor 0 column 1
10 Column 1 1 4 0 r[4]= cursor 1 column 1
11 ResultRow 3 2 0 0 output=r[3..4]
12 Next 2 6 0 0
13 Halt 0 0 0 0
14 Transaction 0 0 3 0 1 usesStmtJournal=0
15 Goto 0 1 0 0
```
Closes #350
2024-10-13 10:11:23 +03:00
jussisaurio
c58b7b42c0
Merge 'Add support for Python 3.13 and drop support for Python 3.8' from Lauri Virtanen
...
The 7th of October 2024 was the first day of Python 3.13 and the last of
Python 3.8 official support.
For reference: https://devguide.python.org/versions/
Closes #367
2024-10-12 17:24:53 +03:00
Lauri Virtanen
aa526f4cb3
Add support for Python 3.13 and drop support for Python 3.8
2024-10-12 15:16:14 +03:00
jussisaurio
04ecbff7fc
Eliminate unnecessary ORDER BY if result set is already ordered
2024-10-10 23:43:39 +03:00
Pekka Enberg
87dd6075f1
Merge 'Fix the Python CI builds' from GV
...
Fixes the Python CI build.
Closes #363
2024-10-10 08:58:17 +03:00
gandeevanr
be94138a5a
Add coverage to development dependencies in pyproject.toml and requirements-dev.txt.
2024-10-09 19:00:53 -07:00
Pekka Enberg
d6829e9794
Merge '[sorter] Hold records in Vec instead of a BTreeMap' from Arpit Saxena
...
We now just insert them one after the other in the vector. When rewind
is called, the vector is sorted. Iterating is just taking elements from
the vector.
Related to #191
SQLite3:
```sh
$ time sqlite3 testing/testing.db "SELECT id FROM users ORDER BY zipcode" > /dev/null
real 0m0.020s
user 0m0.013s
sys 0m0.008s
```
Limbo without this PR:
```sh
$ time target/release/limbo testing/testing.db "SELECT id FROM users ORDER BY zipcode" > /dev/null
real 0m0.285s
user 0m0.257s
sys 0m0.014s
```
Limbo with this PR:
```sh
$ time target/release/limbo testing/testing.db "SELECT id FROM users ORDER BY zipcode" > /dev/null
real 0m0.084s
user 0m0.043s
sys 0m0.032s
```
Closes #362
2024-10-09 16:32:48 +03:00
Arpit Saxena
28a603f56a
[sorter] Hold records in Vec instead of a BTreeMap
...
We now just insert them one after the other in the vector. When rewind
is called, the vector is sorted. Iterating is just taking elements from
the vector
2024-10-09 17:58:28 +05:30
jussisaurio
556f4b73c9
Refine edge case handling: add optional predicate to get_next_record()
2024-10-08 08:23:30 +03:00
jussisaurio
43038cb6aa
Handle seek() edge case with index seek
2024-10-08 07:45:21 +03:00
jussisaurio
572db69b5e
Add TODO comment about index corner case
2024-10-07 17:12:19 +03:00
jussisaurio
93a8110773
dont assume index key has rowid in the second column: its the last
2024-10-07 17:05:38 +03:00
jussisaurio
8563d620af
renaming
2024-10-07 17:03:50 +03:00
jussisaurio
fc71f2b32f
traverse index properly
...
interior index cells have values that are not in the leaves, e.g.
(interior: 3)
/ \
(leaf: 2) (leaf: 4)
so their values need to be emitted after the left subtree is emitted.
2024-10-07 13:04:03 +03:00
Pekka Enberg
0d673710e2
Merge 'Add instr(X,Y) scalar function' from Lauri Virtanen
...
Relates to issue #144
> ## instr(X,Y)
>
> The instr(X,Y) function finds the first occurrence of string Y within
string X and returns the number of prior characters plus 1, or 0 if Y is
nowhere found within X. Or, if X and Y are both BLOBs, then instr(X,Y)
returns one more than the number bytes prior to the first occurrence of
Y, or 0 if Y does not occur anywhere within X. If both arguments X and Y
to instr(X,Y) are non-NULL and are not BLOBs then both are interpreted
as strings. If either X or Y are NULL in instr(X,Y) then the result is
NULL.
Closes #357
2024-10-07 08:59:40 +03:00
jussisaurio
e5cf052f07
Why do sqlite btree child keys have <= keys and not < keys
2024-10-06 23:48:59 +03:00
Lauri Virtanen
0ae1412193
Add instr(X,Y) scalar function
...
Relates to issue #144
2024-10-06 20:19:37 +03:00
jussisaurio
15a66ea662
single seek function in cursor trait
2024-10-06 09:21:15 +03:00
jussisaurio
6e7db36121
reorder
2024-10-06 00:58:32 +03:00
jussisaurio
af9a751d36
Single seek function
2024-10-06 00:56:18 +03:00
jussisaurio
1ae8d28669
Use same move_to() function for tables and indexes
2024-10-06 00:51:14 +03:00
jussisaurio
37f877109e
Reduce duplication in btree.rs
2024-10-06 00:39:50 +03:00
jussisaurio
bb1c8b65e8
fmt
2024-10-06 00:22:12 +03:00
jussisaurio
dde10d2dd7
Better EXPLAIN QUERY PLAN for Operator::Search
2024-10-06 00:19:56 +03:00
jussisaurio
47534cb8df
Get rid of Seekrowid operator in favor of a unified Search operator
2024-10-06 00:11:38 +03:00
