Pere Diaz Bou
cfb6046717
trace instead of debug on debug
2024-11-13 10:14:12 +00:00
Pere Diaz Bou
6834f11e28
add dbug option rust.yml
2024-11-13 10:01:06 +00:00
Pere Diaz Bou
78118bae81
fmt
2024-11-13 09:34:36 +00:00
Pere Diaz Bou
800354144a
remove extra join test
2024-11-12 23:11:04 +00:00
Pere Diaz Bou
eb4105282b
btree: fix btree traversal
2024-11-12 23:10:38 +00:00
Pere Diaz Bou
8d1d2d36cc
core/linux: aligned wal header read
2024-11-12 17:53:59 +00:00
Pere Diaz Bou
19f1a00ca4
bindings/python: update bindings requirements-dev
2024-11-12 17:09:41 +01:00
Pere Diaz Bou
48f0e72e14
checkpoint on drop connection
2024-11-12 17:03:30 +01:00
Pere Diaz Bou
807496a146
core: load pages if not loaded
...
Since pages are now tracked with a single centralized structure, it is
possible for a page to have been unloaded a be kept in memory for
metadata purposes like going back in the stack.
Using the example of going to a parent page which was unloaded for
whatever reason: in this case we need to check if it's loaded, if not,
we load it. Locked still means the same thing, loaded just means the
contents of a page are present in memory and if they are present, they
must be in cache.
2024-11-12 14:44:25 +01:00
Pere Diaz Bou
cd2b61d838
btree: cursor with lineal stack structure
...
Removed MemPage from the code in favor of an array to encode the stack
of the cursor. This is both simpler and better in terms of memory
access.
2024-11-12 10:21:22 +01:00
Pere Diaz Bou
2ce57f6d34
Merge 'Write state machine' from Pere Diaz Bou
...
Returning on ongoing IO requires to save the current state, therefore a
state machine makes sense for write operations. This information is
encoded in WriteInfo/WriteState.
Closes #377
2024-11-12 10:21:00 +01:00
Pere Diaz Bou
ce8378793e
btree: run until finished or IO + bug fixes
2024-11-11 17:03:15 +01:00
Pere Diaz Bou
74974d07e7
clippy fix
2024-11-08 11:14:29 +01:00
Pere Diaz Bou
e5836df160
write state machine
...
Returning on ongoing IO requires to save the current state, therefore a
state machine makes sense for write operations. This information is
encoded in WriteInfo/WriteState.
2024-11-08 11:10:49 +01:00
Pere Diaz Bou
ecfcd8835f
rename get_page -> get_current_page
2024-11-07 15:37:21 +01:00
Pere Diaz Bou
86852935bc
Merge 'Wal' from Pere Diaz Bou
...
This pr introduces first iteration of WAL with automatic checkpointing
of 1000 pages (not configurable right now).
Closes #344
2024-11-07 15:19:47 +01:00
Pere Diaz Bou
7d8fc80f4b
use len instead of size
2024-11-07 14:01:35 +01:00
Pere Diaz Bou
ce1367b76a
move wal creation
2024-11-07 13:51:02 +01:00
Pere Diaz Bou
218c32e0e6
remove unix import in windoews
2024-11-07 13:34:48 +01:00
Pere Diaz Bou
0b46648c0e
write wal header on init
2024-11-07 13:29:57 +01:00
Pere Diaz Bou
0d857661f2
windows complete completion
2024-11-07 09:57:24 +01:00
Pere Diaz Bou
2514287263
windows fix
2024-11-06 18:11:15 +01:00
Pere Diaz Bou
8d9f68ce4e
ignore test wal checkpoint
...
these is flaky now, need to look at it in another time
2024-11-06 18:05:46 +01:00
Pere Diaz Bou
cef78d54de
fix generic
2024-11-06 17:53:14 +01:00
Pere Diaz Bou
df45f17486
fix fmt
2024-11-06 17:47:41 +01:00
Pere Diaz Bou
eb8c462c5f
fix io submission on cacheflush
2024-11-06 16:25:42 +00:00
Pere Diaz Bou
8eb3c89227
wasm,sim fixes
2024-11-05 15:41:30 +01:00
Pere Diaz Bou
a85d599c65
state machine cacheflush
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
da7717edfb
fix generic io match
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
5207e49399
remove extra borrow mut
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
f43e53a815
fix wasm and simulator wal impl
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
0f4270b48f
rebase submit entry
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
70a4ccd8bb
fix linux completion match
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
fc65c5096d
cacheflush state machine
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
129cc1cd6d
fix open_file generic
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
f66e3925f3
fix imports
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
c2453a084d
test wal checkpoint
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
f009eb35c6
suspendable checkpoint
2024-11-05 15:29:54 +01:00
Pere Diaz Bou
c0e51c4ca6
wip wal
2024-11-05 15:29:53 +01:00
Pekka Enberg
0a313c6486
Merge 'Ignore .vscode directory' from Lauri Virtanen
...
Closes #370
2024-10-14 08:04:25 +03:00
Lauri Virtanen
b90a0b3754
Ignore .vscode directory
2024-10-14 00:00:33 +03:00
jussisaurio
7b7dac8ec5
Merge 'clippy fix' from Jussi Saurio
...
Closes #369
2024-10-13 12:21:30 +03:00
jussisaurio
daf5863932
manual fixes based on clippy suggestions
2024-10-13 12:19:04 +03:00
jussisaurio
f634e7f7a3
clippy fix
2024-10-13 12:08:54 +03:00
jussisaurio
bfdd4bc513
Merge 'core/translate: (refactor) use btreetablereference struct instead of tuple' from Jussi Saurio
...
`table` (a `Rc<BTreeTable>`) and `table_identifier` (a `String`) pretty
much always go together in the current implementation, so makes sense to
colocate them as a struct instead of passing around a tuple everywhere
Closes #368
2024-10-13 12:05:19 +03:00
jussisaurio
b3c57d5691
core/translate: (refactor) use btreetablereference struct instead of tuple
2024-10-13 11:25:33 +03:00
Pekka Enberg
d8172ef20b
Update CHANGELOG.md
2024-10-13 10:13:06 +03:00
Pekka Enberg
bf2e254a21
Merge 'core/translate/optimizer: eliminate unnecessary ORDER BY if result set is already ordered' from Jussi Saurio
...
Built on top of, and currently targets, #350 , _not_ main
Closes #365
Examples:
```
limbo> explain select u.first_name, p.name from users u join products p on u.id = p.id order by u.id;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 15 0 0 Start at 15
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 3 0 0 table=p, root=3
4 OpenReadAwait 0 0 0 0
5 RewindAsync 0 0 0 0
6 RewindAwait 0 14 0 0 Rewind table u
7 RowId 0 1 0 0 r[1]=u.rowid
8 SeekRowid 1 1 12 0 if (r[1]!=p.rowid) goto 12
9 Column 0 1 2 0 r[2]=u.first_name
10 Column 1 1 3 0 r[3]=p.name
11 ResultRow 2 2 0 0 output=r[2..3]
12 NextAsync 0 0 0 0
13 NextAwait 0 7 0 0
14 Halt 0 0 0 0
15 Transaction 0 0 0 0
16 Goto 0 1 0 0
```
```
limbo> explain select * from users where age > 80 order by age limit 5;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 23 0 0 Start at 23
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 Integer 80 1 0 0 r[1]=80
6 SeekGT 1 22 1 0
7 DeferredSeek 1 0 0 0
8 RowId 0 2 0 0 r[2]=users.rowid
9 Column 0 1 3 0 r[3]=users.first_name
10 Column 0 2 4 0 r[4]=users.last_name
11 Column 0 3 5 0 r[5]=users.email
12 Column 0 4 6 0 r[6]=users.phone_number
13 Column 0 5 7 0 r[7]=users.address
14 Column 0 6 8 0 r[8]=users.city
15 Column 0 7 9 0 r[9]=users.state
16 Column 0 8 10 0 r[10]=users.zipcode
17 Column 0 9 11 0 r[11]=users.age
18 ResultRow 2 10 0 0 output=r[2..11]
19 DecrJumpZero 12 22 0 0 if (--r[12]==0) goto 22
20 NextAsync 1 0 0 0
21 NextAwait 1 7 0 0
22 Halt 0 0 0 0
23 Transaction 0 0 0 0
24 Integer 5 12 0 0 r[12]=5
25 Goto 0 1 0 0
```
Closes #366
2024-10-13 10:12:28 +03:00
Pekka Enberg
fdf3b0d16b
Merge '(core): Primary key index scans and single-column secondary index scans' from Jussi Saurio
...
This PR adds an index on `users.age` to `testing.db`, and support for
indexed lookups. Only single-column ascending indexes are currently
supported.
This PR also gets rid of `Operator::Seekrowid` in favor of
`Operator::Search` which handles all non-full-table-scan searches: 1.
integer primary key (rowid) point queries 2. integer primary key index
scans, and 3. secondary index scans.
examples:
```
limbo> select first_name, age from users where age > 90 limit 10;
Miranda|90
Sarah|90
Justin|90
Justin|90
John|90
Jeremy|90
Stephanie|90
Joshua|90
Jenny|90
Jennifer|90
limbo> explain query plan select first_name, age from users where age > 90 limit 10;
QUERY PLAN
`--TAKE 10
`--PROJECT first_name, age
| `--SEARCH users USING INDEX age_idx
limbo> explain select first_name, age from users where age > 90 limit 10;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 15 0 0 Start at 15
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 Integer 90 1 0 0 r[1]=90
6 SeekGT 1 14 1 0
7 DeferredSeek 1 0 0 0
8 Column 0 1 2 0 r[2]=users.first_name
9 Column 0 9 3 0 r[3]=users.age
10 ResultRow 2 2 0 0 output=r[2..3]
11 DecrJumpZero 4 14 0 0 if (--r[4]==0) goto 14
12 NextAsync 1 0 0 0
13 NextAwait 1 7 0 0
14 Halt 0 0 0 0
15 Transaction 0 0 0 0
16 Integer 10 4 0 0 r[4]=10
17 Goto 0 1 0 0
```
Sqlite version:
```
sqlite> explain select first_name, age from users where age > 90 limit 10;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 13 0 0 Start at 13
1 Integer 10 1 0 0 r[1]=10; LIMIT counter
2 OpenRead 0 2 0 10 0 root=2 iDb=0; users
3 OpenRead 1 274 0 k(2,,) 0 root=274 iDb=0; age_idx
4 Integer 90 2 0 0 r[2]=90
5 SeekGT 1 12 2 1 0 key=r[2]
6 DeferredSeek 1 0 0 0 Move 0 to 1.rowid if needed
7 Column 0 1 3 0 r[3]= cursor 0 column 1
8 Column 1 0 4 0 r[4]= cursor 1 column 0
9 ResultRow 3 2 0 0 output=r[3..4]
10 DecrJumpZero 1 12 0 0 if (--r[1])==0 goto 12
11 Next 1 6 0 0
12 Halt 0 0 0 0
13 Transaction 0 0 3 0 1 usesStmtJournal=0
14 Goto 0 1 0 0
```
---
´Seek` instructions are also now supported for primary key rowid
searches:
```
limbo> select id, first_name from users where id > 9995;
9996|Donald
9997|Ruth
9998|Dorothy
9999|Gina
10000|Nicole
limbo> explain query plan select id, first_name from users where id > 9995;
QUERY PLAN
`--PROJECT id, first_name
`--SEARCH users USING INTEGER PRIMARY KEY (rowid=?)
limbo> explain select id, first_name from users where id > 9995;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 11 0 0 Start at 11
1 OpenReadAsync 0 2 0 0 table=users, root=2
2 OpenReadAwait 0 0 0 0
3 Integer 9995 1 0 0 r[1]=9995
4 SeekGT 0 10 1 0
5 RowId 0 2 0 0 r[2]=users.rowid
6 Column 0 1 3 0 r[3]=users.first_name
7 ResultRow 2 2 0 0 output=r[2..3]
8 NextAsync 0 0 0 0
9 NextAwait 0 5 0 0
10 Halt 0 0 0 0
11 Transaction 0 0 0 0
12 Goto 0 1 0 0
```
sqlite:
```
sqlite> explain select id, first_name from users where id > 9995;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 8 0 0 Start at 8
1 OpenRead 0 2 0 2 0 root=2 iDb=0; users
2 SeekGT 0 7 1 0 key=r[1]; pk
3 Rowid 0 2 0 0 r[2]=users.rowid
4 Column 0 1 3 0 r[3]= cursor 0 column 1
5 ResultRow 2 2 0 0 output=r[2..3]
6 Next 0 3 0 0
7 Halt 0 0 0 0
8 Transaction 0 0 3 0 1 usesStmtJournal=0
9 Integer 9995 1 0 0 r[1]=9995
10 Goto 0 1 0 0
```
---
More complex example with a join that uses both a rowid lookup and a
secondary index scan:
```
limbo> explain query plan select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
QUERY PLAN
`--PROJECT u.first_name, p.name
`--JOIN
| |--SEARCH u USING INDEX age_idx
| `--SEARCH p USING INTEGER PRIMARY KEY (rowid=?)
limbo> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 18 0 0 Start at 18
1 OpenReadAsync 0 2 0 0 table=u, root=2
2 OpenReadAwait 0 0 0 0
3 OpenReadAsync 1 274 0 0 table=age_idx, root=274
4 OpenReadAwait 0 0 0 0
5 OpenReadAsync 2 3 0 0 table=p, root=3
6 OpenReadAwait 0 0 0 0
7 Integer 70 1 0 0 r[1]=70
8 SeekGT 1 17 1 0
9 DeferredSeek 1 0 0 0
10 RowId 0 2 0 0 r[2]=u.rowid
11 SeekRowid 2 2 15 0 if (r[2]!=p.rowid) goto 15
12 Column 0 1 3 0 r[3]=u.first_name
13 Column 2 1 4 0 r[4]=p.name
14 ResultRow 3 2 0 0 output=r[3..4]
15 NextAsync 1 0 0 0
16 NextAwait 1 9 0 0
17 Halt 0 0 0 0
18 Transaction 0 0 0 0
19 Goto 0 1 0 0
```
sqlite:
```
sqlite> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 14 0 0 Start at 14
1 OpenRead 0 2 0 10 0 root=2 iDb=0; users
2 OpenRead 2 274 0 k(2,,) 0 root=274 iDb=0; age_idx
3 OpenRead 1 3 0 2 0 root=3 iDb=0; products
4 Integer 70 1 0 0 r[1]=70
5 SeekGT 2 13 1 1 0 key=r[1]
6 DeferredSeek 2 0 0 0 Move 0 to 2.rowid if needed
7 IdxRowid 2 2 0 0 r[2]=rowid; users.rowid
8 SeekRowid 1 12 2 0 intkey=r[2]
9 Column 0 1 3 0 r[3]= cursor 0 column 1
10 Column 1 1 4 0 r[4]= cursor 1 column 1
11 ResultRow 3 2 0 0 output=r[3..4]
12 Next 2 6 0 0
13 Halt 0 0 0 0
14 Transaction 0 0 3 0 1 usesStmtJournal=0
15 Goto 0 1 0 0
```
Closes #350
2024-10-13 10:11:23 +03:00
jussisaurio
c58b7b42c0
Merge 'Add support for Python 3.13 and drop support for Python 3.8' from Lauri Virtanen
...
The 7th of October 2024 was the first day of Python 3.13 and the last of
Python 3.8 official support.
For reference: https://devguide.python.org/versions/
Closes #367
2024-10-12 17:24:53 +03:00
