When the python tests fail, they will sometimes truncate the output if it is smaller than the `PIPE_BUFF` size. With this fix we can now properly print the backtrace, when the program panics. # Before This is the problematic CI output from #1331 that led me to fix this. In this case, it was already truncating the output of the `assert` prints. ``` ./testing/cli_tests/extensions.py Extension ./target/debug/liblimbo_regexp loaded successfully. Testing: uuid functions are registered properly with ext loaded Testing: scalar alias's are registered properly Testing: median agg function returns null when ext not loaded Testing: median agg function works Testing: median agg function works with odd number of elements Testing: test aggregate percentile function with 2 arguments works Testing: test aggregate percentile function with 1 argument works Testing: crypto_blake3 returns null when ext not loaded Testing: blake3 should encrypt correctly Testing: md5 should encrypt correctly Testing: sha1 should encrypt correctly Testing: sha256 should encrypt correctly Testing: sha384 should encrypt correctly Testing: sha512 should encrypt correctly Testing: base32 should encode correctly Testing: base32 should decode correctly Testing: base64 should encode correctly Testing: base64 should decode correctly Testing: base85 should encode correctly Testing: base85 should decode correctly Testing: hex should encode correctly Testing: hex should decode correctly Testing: url should encode correctly Testing: url should decode correctly Testing: ipfamily function returns null when ext not loaded Testing: ipfamily function returns 4 for IPv4 Testing: ipfamily function returns 6 for IPv6 Testing: ipcontains function returns 1 for IPv4 Testing: ipcontains function returns 0 for IPv4 Testing: iphost function returns the host for IPv4 Testing: iphost function returns the host for IPv6 Testing: ipmasklen function returns the mask length for IPv4 Testing: ipmasklen function returns the mask length for IPv6 Testing: ipnetwork function returns the flattened CIDR for IPv4 Testing: ipnetwork function returns the network for IPv6 Testing: testvfs not loaded Testing: testvfs extension loaded thread 'main' panicked at core/storage/pager.rs:61:38: called `Option::unwrap()` on a `None` value note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace Test FAILED: Error encountered in Limbo shell. make: *** [Makefile:70: test-extensions] Error 1 ``` # After ``` with-env {RUST_BACKTRACE:1} {make test-extensions} cargo build Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.51s cargo build --package limbo_regexp Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp) Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.19s ./testing/cli_tests/extensions.py Extension ./target/debug/liblimbo_regexp loaded successfully. Testing: uuid functions are registered properly with ext loaded Testing: scalar alias's are registered properly Testing: median agg function returns null when ext not loaded Testing: median agg function works Testing: median agg function works with odd number of elements Testing: test aggregate percentile function with 2 arguments works Testing: test aggregate percentile function with 1 argument works Testing: crypto_blake3 returns null when ext not loaded Testing: blake3 should encrypt correctly Testing: md5 should encrypt correctly Testing: sha1 should encrypt correctly Testing: sha256 should encrypt correctly Testing: sha384 should encrypt correctly Testing: sha512 should encrypt correctly Testing: base32 should encode correctly Testing: base32 should decode correctly Testing: base64 should encode correctly Testing: base64 should decode correctly Testing: base85 should encode correctly Testing: base85 should decode correctly Testing: hex should encode correctly Testing: hex should decode correctly Testing: url should encode correctly Testing: url should decode correctly Testing: ipfamily function returns null when ext not loaded Testing: ipfamily function returns 4 for IPv4 Testing: ipfamily function returns 6 for IPv6 Testing: ipcontains function returns 1 for IPv4 Testing: ipcontains function returns 0 for IPv4 Testing: iphost function returns the host for IPv4 Testing: iphost function returns the host for IPv6 Testing: ipmasklen function returns the mask length for IPv4 Testing: ipmasklen function returns the mask length for IPv6 Testing: ipnetwork function returns the flattened CIDR for IPv4 Testing: ipnetwork function returns the network for IPv6 Testing: testvfs not loaded Testing: testvfs extension loaded thread 'main' panicked at core/storage/pager.rs:61:38: called `Option::unwrap()` on a `None` value stack backtrace: 0: rust_begin_unwind at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5 1: core::panicking::panic_fmt at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14 2: core::panicking::panic at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:148:5 3: core::option::unwrap_failed at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/option.rs:2012:5 4: core::option::Option<T>::unwrap at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:972:21 5: limbo_core::storage:📟:Page::get_contents at ./core/storage/pager.rs:61:9 6: limbo_core::storage::btree::BTreeCursor::balance_non_root at ./core/storage/btree.rs:1723:40 7: limbo_core::storage::btree::BTreeCursor::balance at ./core/storage/btree.rs:1570:35 8: limbo_core::storage::btree::BTreeCursor::insert_into_page at ./core/storage/btree.rs:1512:35 9: limbo_core::storage::btree::BTreeCursor::insert at ./core/storage/btree.rs:3024:31 10: limbo_core::vdbe::execute::op_insert at ./core/vdbe/execute.rs:3654:23 11: limbo_core::vdbe::Program::step at ./core/vdbe/mod.rs:379:23 12: limbo_core::Statement::step at ./core/lib.rs:582:9 13: limbo::app::Limbo::print_query_result at ./cli/app.rs:657:27 14: limbo::app::Limbo::run_query at ./cli/app.rs:420:20 15: limbo::app::Limbo::handle_input_line at ./cli/app.rs:527:13 16: limbo::main at ./cli/main.rs:29:31 17: core::ops::function::FnOnce::call_once at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. Testing: Tested large write to testfs Test FAILED: Test failed SQL: SELECT count(*) FROM test; Actual: None make: *** [test-extensions] Error 1 ``` Closes #1346
Project Limbo
Limbo is a project to build the modern evolution of SQLite.
Features and Roadmap
Limbo is a work-in-progress, in-process OLTP database engine library written in Rust that has:
- Asynchronous I/O support on Linux with
io_uring - SQLite compatibility [doc] for SQL dialect, file formats, and the C API
- Language bindings for JavaScript/WebAssembly, Rust, Go, Python, and Java
- OS support for Linux, macOS, and Windows
In the future, we will be also working on:
- Integrated vector search for embeddings and vector similarity.
BEGIN CONCURRENTfor improved write throughput.- Improved schema management including better
ALTERsupport and strict column types by default.
Getting Started
💻 Command Line
You can install the latest `limbo` release with:
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/tursodatabase/limbo/releases/latest/download/limbo_cli-installer.sh | sh
Then launch the shell to execute SQL statements:
Limbo
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database
limbo> CREATE TABLE users (id INT PRIMARY KEY, username TEXT);
limbo> INSERT INTO users VALUES (1, 'alice');
limbo> INSERT INTO users VALUES (2, 'bob');
limbo> SELECT * FROM users;
1|alice
2|bob
You can also build and run the latest development version with:
cargo run
✨ JavaScript
npm i limbo-wasm
Example usage:
import { Database } from 'limbo-wasm';
const db = new Database('sqlite.db');
const stmt = db.prepare('SELECT * FROM users');
const users = stmt.all();
console.log(users);
🐍 Python
pip install pylimbo
Example usage:
import limbo
con = limbo.connect("sqlite.db")
cur = con.cursor()
res = cur.execute("SELECT * FROM users")
print(res.fetchone())
🐹 Go
- Clone the repository
- Build the library and set your LD_LIBRARY_PATH to include limbo's target directory
cargo build --package limbo-go
export LD_LIBRARY_PATH=/path/to/limbo/target/debug:$LD_LIBRARY_PATH
- Use the driver
go get github.com/tursodatabase/limbo
go install github.com/tursodatabase/limbo
Example usage:
import (
"database/sql"
_"github.com/tursodatabase/limbo"
)
conn, _ = sql.Open("sqlite3", "sqlite.db")
defer conn.Close()
stmt, _ := conn.Prepare("select * from users")
defer stmt.Close()
rows, _ = stmt.Query()
for rows.Next() {
var id int
var username string
_ := rows.Scan(&id, &username)
fmt.Printf("User: ID: %d, Username: %s\n", id, username)
}
☕️ Java
We integrated Limbo into JDBC. For detailed instructions on how to use Limbo with java, please refer to the README.md under bindings/java.
Contributing
We'd love to have you contribute to Limbo! Please check out the contribution guide to get started.
FAQ
How is Limbo different from Turso's libSQL?
Limbo is a project to build the modern evolution of SQLite in Rust, with a strong open contribution focus and features like native async support, vector search, and more. The libSQL project is also an attempt to evolve SQLite in a similar direction, but through a fork rather than a rewrite.
Rewriting SQLite in Rust started as an unassuming experiment, and due to its incredible success, replaces libSQL as our intended direction. At this point, libSQL is production ready, Limbo is not - although it is evolving rapidly. As the project starts to near production readiness, we plan to rename it to just "Turso". More details here.
Publications
- Pekka Enberg, Sasu Tarkoma, Jon Crowcroft Ashwin Rao (2024). Serverless Runtime / Database Co-Design With Asynchronous I/O. In EdgeSys ‘24. [PDF]
- Pekka Enberg, Sasu Tarkoma, and Ashwin Rao (2023). Towards Database and Serverless Runtime Co-Design. In CoNEXT-SW ’23. [PDF] [Slides]
License
This project is licensed under the MIT license.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Limbo by you, shall be licensed as MIT, without any additional terms or conditions.
Contributors
Thanks to all the contributors to Limbo!
