mirror of
https://github.com/aljazceru/turso.git
synced 2025-12-24 19:44:21 +01:00
fdf3b0d16b
This PR adds an index on `users.age` to `testing.db`, and support for indexed lookups. Only single-column ascending indexes are currently supported. This PR also gets rid of `Operator::Seekrowid` in favor of `Operator::Search` which handles all non-full-table-scan searches: 1. integer primary key (rowid) point queries 2. integer primary key index scans, and 3. secondary index scans. examples: ``` limbo> select first_name, age from users where age > 90 limit 10; Miranda|90 Sarah|90 Justin|90 Justin|90 John|90 Jeremy|90 Stephanie|90 Joshua|90 Jenny|90 Jennifer|90 limbo> explain query plan select first_name, age from users where age > 90 limit 10; QUERY PLAN `--TAKE 10 `--PROJECT first_name, age | `--SEARCH users USING INDEX age_idx limbo> explain select first_name, age from users where age > 90 limit 10; addr opcode p1 p2 p3 p4 p5 comment ---- ----------------- ---- ---- ---- ------------- -- ------- 0 Init 0 15 0 0 Start at 15 1 OpenReadAsync 0 2 0 0 table=users, root=2 2 OpenReadAwait 0 0 0 0 3 OpenReadAsync 1 274 0 0 table=age_idx, root=274 4 OpenReadAwait 0 0 0 0 5 Integer 90 1 0 0 r[1]=90 6 SeekGT 1 14 1 0 7 DeferredSeek 1 0 0 0 8 Column 0 1 2 0 r[2]=users.first_name 9 Column 0 9 3 0 r[3]=users.age 10 ResultRow 2 2 0 0 output=r[2..3] 11 DecrJumpZero 4 14 0 0 if (--r[4]==0) goto 14 12 NextAsync 1 0 0 0 13 NextAwait 1 7 0 0 14 Halt 0 0 0 0 15 Transaction 0 0 0 0 16 Integer 10 4 0 0 r[4]=10 17 Goto 0 1 0 0 ``` Sqlite version: ``` sqlite> explain select first_name, age from users where age > 90 limit 10; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 13 0 0 Start at 13 1 Integer 10 1 0 0 r[1]=10; LIMIT counter 2 OpenRead 0 2 0 10 0 root=2 iDb=0; users 3 OpenRead 1 274 0 k(2,,) 0 root=274 iDb=0; age_idx 4 Integer 90 2 0 0 r[2]=90 5 SeekGT 1 12 2 1 0 key=r[2] 6 DeferredSeek 1 0 0 0 Move 0 to 1.rowid if needed 7 Column 0 1 3 0 r[3]= cursor 0 column 1 8 Column 1 0 4 0 r[4]= cursor 1 column 0 9 ResultRow 3 2 0 0 output=r[3..4] 10 DecrJumpZero 1 12 0 0 if (--r[1])==0 goto 12 11 Next 1 6 0 0 12 Halt 0 0 0 0 13 Transaction 0 0 3 0 1 usesStmtJournal=0 14 Goto 0 1 0 0 ``` --- ´Seek` instructions are also now supported for primary key rowid searches: ``` limbo> select id, first_name from users where id > 9995; 9996|Donald 9997|Ruth 9998|Dorothy 9999|Gina 10000|Nicole limbo> explain query plan select id, first_name from users where id > 9995; QUERY PLAN `--PROJECT id, first_name `--SEARCH users USING INTEGER PRIMARY KEY (rowid=?) limbo> explain select id, first_name from users where id > 9995; addr opcode p1 p2 p3 p4 p5 comment ---- ----------------- ---- ---- ---- ------------- -- ------- 0 Init 0 11 0 0 Start at 11 1 OpenReadAsync 0 2 0 0 table=users, root=2 2 OpenReadAwait 0 0 0 0 3 Integer 9995 1 0 0 r[1]=9995 4 SeekGT 0 10 1 0 5 RowId 0 2 0 0 r[2]=users.rowid 6 Column 0 1 3 0 r[3]=users.first_name 7 ResultRow 2 2 0 0 output=r[2..3] 8 NextAsync 0 0 0 0 9 NextAwait 0 5 0 0 10 Halt 0 0 0 0 11 Transaction 0 0 0 0 12 Goto 0 1 0 0 ``` sqlite: ``` sqlite> explain select id, first_name from users where id > 9995; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 8 0 0 Start at 8 1 OpenRead 0 2 0 2 0 root=2 iDb=0; users 2 SeekGT 0 7 1 0 key=r[1]; pk 3 Rowid 0 2 0 0 r[2]=users.rowid 4 Column 0 1 3 0 r[3]= cursor 0 column 1 5 ResultRow 2 2 0 0 output=r[2..3] 6 Next 0 3 0 0 7 Halt 0 0 0 0 8 Transaction 0 0 3 0 1 usesStmtJournal=0 9 Integer 9995 1 0 0 r[1]=9995 10 Goto 0 1 0 0 ``` --- More complex example with a join that uses both a rowid lookup and a secondary index scan: ``` limbo> explain query plan select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70; QUERY PLAN `--PROJECT u.first_name, p.name `--JOIN | |--SEARCH u USING INDEX age_idx | `--SEARCH p USING INTEGER PRIMARY KEY (rowid=?) limbo> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70; addr opcode p1 p2 p3 p4 p5 comment ---- ----------------- ---- ---- ---- ------------- -- ------- 0 Init 0 18 0 0 Start at 18 1 OpenReadAsync 0 2 0 0 table=u, root=2 2 OpenReadAwait 0 0 0 0 3 OpenReadAsync 1 274 0 0 table=age_idx, root=274 4 OpenReadAwait 0 0 0 0 5 OpenReadAsync 2 3 0 0 table=p, root=3 6 OpenReadAwait 0 0 0 0 7 Integer 70 1 0 0 r[1]=70 8 SeekGT 1 17 1 0 9 DeferredSeek 1 0 0 0 10 RowId 0 2 0 0 r[2]=u.rowid 11 SeekRowid 2 2 15 0 if (r[2]!=p.rowid) goto 15 12 Column 0 1 3 0 r[3]=u.first_name 13 Column 2 1 4 0 r[4]=p.name 14 ResultRow 3 2 0 0 output=r[3..4] 15 NextAsync 1 0 0 0 16 NextAwait 1 9 0 0 17 Halt 0 0 0 0 18 Transaction 0 0 0 0 19 Goto 0 1 0 0 ``` sqlite: ``` sqlite> explain select u.first_name, p.name from users u join products p on u.id = p.id and u.age > 70; addr opcode p1 p2 p3 p4 p5 comment ---- ------------- ---- ---- ---- ------------- -- ------------- 0 Init 0 14 0 0 Start at 14 1 OpenRead 0 2 0 10 0 root=2 iDb=0; users 2 OpenRead 2 274 0 k(2,,) 0 root=274 iDb=0; age_idx 3 OpenRead 1 3 0 2 0 root=3 iDb=0; products 4 Integer 70 1 0 0 r[1]=70 5 SeekGT 2 13 1 1 0 key=r[1] 6 DeferredSeek 2 0 0 0 Move 0 to 2.rowid if needed 7 IdxRowid 2 2 0 0 r[2]=rowid; users.rowid 8 SeekRowid 1 12 2 0 intkey=r[2] 9 Column 0 1 3 0 r[3]= cursor 0 column 1 10 Column 1 1 4 0 r[4]= cursor 1 column 1 11 ResultRow 3 2 0 0 output=r[3..4] 12 Next 2 6 0 0 13 Halt 0 0 0 0 14 Transaction 0 0 3 0 1 usesStmtJournal=0 15 Goto 0 1 0 0 ``` Closes #350
Limbo
Limbo is a work-in-progress, in-process OLTP database management system, compatible with SQLite.
Features
- In-process OLTP database engine library
- Asynchronous I/O support with
io_uring - SQLite compatibility (status)
- SQL dialect support
- File format support
- SQLite C API
- JavaScript/WebAssembly bindings (wip)
Getting Started
Installing:
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/penberg/limbo/releases/latest/download/limbo-installer.sh | sh
Limbo is currently work-in-progress so it's recommended that you either use the sqlite3 program to create a test database:
$ sqlite3 database.db
SQLite version 3.42.0 2023-05-16 12:36:15
Enter ".help" for usage hints.
sqlite> CREATE TABLE users (id INT PRIMARY KEY, username TEXT);
sqlite> INSERT INTO users VALUES (1, 'alice');
sqlite> INSERT INTO users VALUES (2, 'bob');
or use the testing script to generate one for you:
pipenv run ./testing/gen-database.py
You can then start the Limbo shell with:
$ limbo database.db
Welcome to Limbo SQL shell!
> SELECT * FROM users LIMIT 1;
|1|Cody|Miller|mhurst@example.org|525.595.7319x21268|33667 Shaw Extension Suite 104|West Robert|VA|45161|`
Developing
Run tests:
cargo test
Test coverage report:
cargo tarpaulin -o html
Run benchmarks:
cargo bench
Run benchmarks and generate flamegraphs:
echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid
cargo bench --bench benchmark -- --profile-time=5
FAQ
How is Limbo different from libSQL?
Limbo is a research project to build a SQLite compatible in-process database in Rust with native async support. The libSQL project, on the other hand, is an open source, open contribution fork of SQLite, with focus on production features such as replication, backups, encryption, and so on. There is no hard dependency between the two projects. Of course, if Limbo becomes widely successful, we might consider merging with libSQL, but that is something that will be decided in the future.
Publications
- Pekka Enberg, Sasu Tarkoma, Jon Crowcroft Ashwin Rao (2024). Serverless Runtime / Database Co-Design With Asynchronous I/O. In EdgeSys ‘24. [PDF]
- Pekka Enberg, Sasu Tarkoma, and Ashwin Rao (2023). Towards Database and Serverless Runtime Co-Design. In CoNEXT-SW ’23. [PDF] [Slides]
Contributing
We'd love to have you contribute to Limbo! Check out the contribution guide to get started.
License
This project is licensed under the MIT license.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Limbo by you, shall be licensed as MIT, without any additional terms or conditions.