# Multi column indexes + index seek refactor ## PR reader guide I would say mostly you should just focus on the content of `optimizer.rs` and `plan.rs` because the rest is just small type changes, or in the case of `main_loop.rs`, a bunch of logic was just moved out of there and rewritten. ## New feature - multi column index seeks This PR adds support for utilizing multi-column indexes properly, i.e. using as many columns in the seek key as possible. Previously, we only used max one column per index. I've modified the existing compound index seek fuzz test to use this functionality. ## Refactoring of index seek related logic This PR moves a lot of index seek related logic out of `main_loop.rs` into `optimizer.rs` and `plan.rs` and introduces a bunch of helper structures to model finding and using an index to perform a seek + scan. ## Examples Here are some examples of multi-column seeks: ### Example table setup: ```sql sqlite> CREATE TABLE t(a,b,c,d,e); sqlite> CREATE INDEX abc ON t (a,b,c); -- create 10000 rows with random values between 0-9 for all columns sqlite >INSERT INTO t SELECT ABS(RANDOM() % 10),ABS(RANDOM() % 10),ABS(RANDOM() % 10),ABS(RANDOM() % 10),ABS(RANDOM() % 10) FROM generate_series(1,10000,1); ``` ### Example bytecode plans, results and timings vs main branch: ```sql limbo> EXPLAIN SELECT * FROM t WHERE a = 5 and b = 6 and c = 7; addr opcode p1 p2 p3 p4 p5 comment ---- ----------------- ---- ---- ---- ------------- -- ------- 0 Init 0 20 0 0 Start at 20 1 OpenReadAsync 0 2 0 0 table=t, root=2 2 OpenReadAwait 0 0 0 0 3 OpenReadAsync 1 3 0 0 table=abc, root=3 4 OpenReadAwait 0 0 0 0 5 Integer 5 6 0 0 r[6]=5 6 Integer 6 7 0 0 r[7]=6 7 Integer 7 8 0 0 r[8]=7 8 SeekGE 1 19 6 0 key=[6..8] 9 IdxGT 1 19 6 0 key=[6..8] 10 DeferredSeek 1 0 0 0 11 Column 0 0 1 0 r[1]=t.a 12 Column 0 1 2 0 r[2]=t.b 13 Column 0 2 3 0 r[3]=t.c 14 Column 0 3 4 0 r[4]=t.d 15 Column 0 4 5 0 r[5]=t.e 16 ResultRow 1 5 0 0 output=r[1..5] 17 NextAsync 1 0 0 0 18 NextAwait 1 9 0 0 19 Halt 0 0 0 0 20 Transaction 0 0 0 0 write=false 21 Goto 0 1 0 0 limbo> SELECT * FROM t WHERE a = 5 and b = 6 and c = 7; 5|6|7|9|9 5|6|7|4|7 5|6|7|3|2 5|6|7|3|7 5|6|7|5|2 5|6|7|5|3 5|6|7|9|7 runtime (debug build, this branch): total: 2 ms (this includes parsing/coloring of cli app) runtime (debud build, main branch): total: 67 ms (this includes parsing/coloring of cli app) ``` ```sql limbo> EXPLAIN SELECT * FROM t WHERE a = 5 and b = 6 and c < 7; addr opcode p1 p2 p3 p4 p5 comment ---- ----------------- ---- ---- ---- ------------- -- ------- 0 Init 0 21 0 0 Start at 21 1 OpenReadAsync 0 2 0 0 table=t, root=2 2 OpenReadAwait 0 0 0 0 3 OpenReadAsync 1 3 0 0 table=abc, root=3 4 OpenReadAwait 0 0 0 0 5 Integer 5 6 0 0 r[6]=5 6 Integer 6 7 0 0 r[7]=6 7 Null 0 8 0 0 r[8]=NULL 8 SeekGT 1 20 6 0 key=[6..8] 9 Integer 7 8 0 0 r[8]=7 10 IdxGE 1 20 6 0 key=[6..8] 11 DeferredSeek 1 0 0 0 12 Column 0 0 1 0 r[1]=t.a 13 Column 0 1 2 0 r[2]=t.b 14 Column 0 2 3 0 r[3]=t.c 15 Column 0 3 4 0 r[4]=t.d 16 Column 0 4 5 0 r[5]=t.e 17 ResultRow 1 5 0 0 output=r[1..5] 18 NextAsync 1 0 0 0 19 NextAwait 1 10 0 0 20 Halt 0 0 0 0 21 Transaction 0 0 0 0 write=false 22 Goto 0 1 0 0 limbo> SELECT * FROM t WHERE a = 5 and b = 6 and c < 7; 5|6|0|0|3 5|6|0|5|1 5|6|0|3|1 5|6|0|6|3 5|6|0|8|1 5|6|0|2|7 5|6|0|9|9 5|6|0|5|3 5|6|0|4|2 5|6|0|4|2 5|6|0|0|2 5|6|0|7|2 5|6|1|8|5 5|6|1|7|5 5|6|1|7|2 5|6|1|1|2 5|6|1|6|5 5|6|1|1|5 5|6|1|5|7 5|6|1|1|9 5|6|1|4|3 5|6|1|1|2 5|6|1|2|2 5|6|1|4|4 5|6|1|9|6 5|6|1|2|5 5|6|1|2|4 5|6|1|7|1 5|6|2|0|9 5|6|2|6|9 5|6|2|4|5 5|6|2|9|3 5|6|2|5|2 5|6|2|9|0 5|6|2|7|1 5|6|3|6|5 5|6|3|8|5 5|6|3|5|4 5|6|3|5|2 5|6|3|1|1 5|6|3|2|0 5|6|3|9|3 5|6|3|6|9 5|6|3|7|6 5|6|3|3|5 5|6|3|0|8 5|6|3|6|4 5|6|4|1|1 5|6|4|9|8 5|6|4|3|7 5|6|4|1|3 5|6|4|8|9 5|6|4|9|7 5|6|4|7|9 5|6|4|8|8 5|6|4|3|1 5|6|4|2|6 5|6|4|5|7 5|6|4|2|6 5|6|4|4|3 5|6|5|2|4 5|6|5|6|7 5|6|5|3|8 5|6|5|7|8 5|6|5|9|6 5|6|5|2|7 5|6|5|1|7 5|6|5|0|6 5|6|6|2|4 5|6|6|9|4 5|6|6|4|9 5|6|6|5|6 5|6|6|2|2 5|6|6|0|6 runtime (debug build, this branch): total: 9 ms (this includes parsing/coloring of cli app) runtime (debug build, main branch): total: 71 ms (this includes parsing/coloring of cli app) ``` ```sql limbo> EXPLAIN SELECT * FROM t WHERE a = 5 and b = 6 and c < 7 ORDER BY a desc, b desc, c desc; addr opcode p1 p2 p3 p4 p5 comment ---- ----------------- ---- ---- ---- ------------- -- ------- 0 Init 0 20 0 0 Start at 20 1 OpenReadAsync 0 2 0 0 table=t, root=2 2 OpenReadAwait 0 0 0 0 3 OpenReadAsync 1 3 0 0 table=abc, root=3 4 OpenReadAwait 0 0 0 0 5 Integer 5 6 0 0 r[6]=5 6 Integer 6 7 0 0 r[7]=6 7 Integer 7 8 0 0 r[8]=7 8 SeekLT 1 19 6 0 key=[6..8] 9 IdxLT 1 19 6 0 key=[6..7] 10 DeferredSeek 1 0 0 0 11 Column 0 0 1 0 r[1]=t.a 12 Column 0 1 2 0 r[2]=t.b 13 Column 0 2 3 0 r[3]=t.c 14 Column 0 3 4 0 r[4]=t.d 15 Column 0 4 5 0 r[5]=t.e 16 ResultRow 1 5 0 0 output=r[1..5] 17 PrevAsync 1 0 0 0 18 PrevAwait 1 0 0 0 19 Halt 0 0 0 0 20 Transaction 0 0 0 0 write=false 21 Goto 0 1 0 0 limbo> SELECT * FROM t WHERE a = 5 and b = 6 and c < 7 ORDER BY a desc, b desc, c desc; 5|6|6|0|6 5|6|6|2|2 5|6|6|5|6 5|6|6|4|9 5|6|6|9|4 5|6|6|2|4 5|6|5|0|6 5|6|5|1|7 5|6|5|2|7 5|6|5|9|6 5|6|5|7|8 5|6|5|3|8 5|6|5|6|7 5|6|5|2|4 5|6|4|4|3 5|6|4|2|6 5|6|4|5|7 5|6|4|2|6 5|6|4|3|1 5|6|4|8|8 5|6|4|7|9 5|6|4|9|7 5|6|4|8|9 5|6|4|1|3 5|6|4|3|7 5|6|4|9|8 5|6|4|1|1 5|6|3|6|4 5|6|3|0|8 5|6|3|3|5 5|6|3|7|6 5|6|3|6|9 5|6|3|9|3 5|6|3|2|0 5|6|3|1|1 5|6|3|5|2 5|6|3|5|4 5|6|3|8|5 5|6|3|6|5 5|6|2|7|1 5|6|2|9|0 5|6|2|5|2 5|6|2|9|3 5|6|2|4|5 5|6|2|6|9 5|6|2|0|9 5|6|1|7|1 5|6|1|2|4 5|6|1|2|5 5|6|1|9|6 5|6|1|4|4 5|6|1|2|2 5|6|1|1|2 5|6|1|4|3 5|6|1|1|9 5|6|1|5|7 5|6|1|1|5 5|6|1|6|5 5|6|1|1|2 5|6|1|7|2 5|6|1|7|5 5|6|1|8|5 5|6|0|7|2 5|6|0|0|2 5|6|0|4|2 5|6|0|4|2 5|6|0|5|3 5|6|0|9|9 5|6|0|2|7 5|6|0|8|1 5|6|0|6|3 5|6|0|3|1 5|6|0|5|1 5|6|0|0|3 runtime (debug build, this branch): total: 9 ms (this includes parsing/coloring of cli app) runtime (debug build, main branch): total: 71 ms (this includes parsing/coloring of cli app) ``` Closes #1288
Project Limbo
Limbo is a project to build the modern evolution of SQLite.
Features and Roadmap
Limbo is a work-in-progress, in-process OLTP database engine library written in Rust that has:
- Asynchronous I/O support on Linux with
io_uring - SQLite compatibility [doc] for SQL dialect, file formats, and the C API
- Language bindings for JavaScript/WebAssembly, Rust, Go, Python, and Java
- OS support for Linux, macOS, and Windows
In the future, we will be also working on:
- Integrated vector search for embeddings and vector similarity.
BEGIN CONCURRENTfor improved write throughput.- Improved schema management including better
ALTERsupport and strict column types by default.
Getting Started
💻 Command Line
You can install the latest `limbo` release with:
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/tursodatabase/limbo/releases/latest/download/limbo_cli-installer.sh | sh
Then launch the shell to execute SQL statements:
Limbo
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database
limbo> CREATE TABLE users (id INT PRIMARY KEY, username TEXT);
limbo> INSERT INTO users VALUES (1, 'alice');
limbo> INSERT INTO users VALUES (2, 'bob');
limbo> SELECT * FROM users;
1|alice
2|bob
You can also build and run the latest development version with:
cargo run
✨ JavaScript
npm i limbo-wasm
Example usage:
import { Database } from 'limbo-wasm';
const db = new Database('sqlite.db');
const stmt = db.prepare('SELECT * FROM users');
const users = stmt.all();
console.log(users);
🐍 Python
pip install pylimbo
Example usage:
import limbo
con = limbo.connect("sqlite.db")
cur = con.cursor()
res = cur.execute("SELECT * FROM users")
print(res.fetchone())
🐹 Go
- Clone the repository
- Build the library and set your LD_LIBRARY_PATH to include limbo's target directory
cargo build --package limbo-go
export LD_LIBRARY_PATH=/path/to/limbo/target/debug:$LD_LIBRARY_PATH
- Use the driver
go get github.com/tursodatabase/limbo
go install github.com/tursodatabase/limbo
Example usage:
import (
"database/sql"
_"github.com/tursodatabase/limbo"
)
conn, _ = sql.Open("sqlite3", "sqlite.db")
defer conn.Close()
stmt, _ := conn.Prepare("select * from users")
defer stmt.Close()
rows, _ = stmt.Query()
for rows.Next() {
var id int
var username string
_ := rows.Scan(&id, &username)
fmt.Printf("User: ID: %d, Username: %s\n", id, username)
}
☕️ Java
We integrated Limbo into JDBC. For detailed instructions on how to use Limbo with java, please refer to the README.md under bindings/java.
Contributing
We'd love to have you contribute to Limbo! Please check out the contribution guide to get started.
FAQ
How is Limbo different from Turso's libSQL?
Limbo is a project to build the modern evolution of SQLite in Rust, with a strong open contribution focus and features like native async support, vector search, and more. The libSQL project is also an attempt to evolve SQLite in a similar direction, but through a fork rather than a rewrite.
Rewriting SQLite in Rust started as an unassuming experiment, and due to its incredible success, replaces libSQL as our intended direction. At this point, libSQL is production ready, Limbo is not - although it is evolving rapidly. As the project starts to near production readiness, we plan to rename it to just "Turso". More details here.
Publications
- Pekka Enberg, Sasu Tarkoma, Jon Crowcroft Ashwin Rao (2024). Serverless Runtime / Database Co-Design With Asynchronous I/O. In EdgeSys ‘24. [PDF]
- Pekka Enberg, Sasu Tarkoma, and Ashwin Rao (2023). Towards Database and Serverless Runtime Co-Design. In CoNEXT-SW ’23. [PDF] [Slides]
License
This project is licensed under the MIT license.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Limbo by you, shall be licensed as MIT, without any additional terms or conditions.
Contributors
Thanks to all the contributors to Limbo!
