mirror of
https://github.com/aljazceru/turso.git
synced 2025-12-26 20:44:23 +01:00
1e4e8c243a
Closes #2225. ## What We currently do not use pages in the [freelist](https://www.sqlite.org/fileformat.html#the_freelist) at all when allocating new pages. ## Why is this bad The effect of this is that 1. UPDATEs with overflow pages become really slow and 2. the database size grows really quickly. See #2225 for an extreme example comparison with SQLite. ## The fix Whenever `allocate_page()` is called, we first check if we have pages in the freelist, and if we do, we recycle one of those pages instead of creating a new one. If there are no freelist pages, we allocate a new page as normal. ## Implementation notes - `allocate_page()` now needs to return an `IOResult`, which means all of its callers also need to return an `IOResult`, necessitating quite a bit of new state machine logic to ensure re-entrancy. - I left a few "synchronous IO hacks" in the `balance()` routine because the size of this PR would balloon even more than it already has if I were to fix those immediately in this PR. - `fill_cell_payload()` uses some `unsafe` code to avoid lifetime issues, and adds an unfortunate double-indirection via `Arc<Mutex<Vec<T>>>` because the existing btree code constantly clones `WriteState`, and we must ensure the underlying buffers referenced by raw pointers in `fill_cell_payload` remain valid. **Follow-up cleanups:** 1. remove synchronous IO hacks that would require even more state machines and are best left for another PR 2. remove `Clone` from `WriteState` and implement it better ## Perf comparison `main`: 33 seconds ``` jussi@Jussis-MacBook-Pro limbo % time target/release/tursodb --experimental-indexes apinatest_main.db <<'EOF' create table t(x, y, z unique); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); EOF Turso v0.1.3-pre.3 Enter ".help" for usage hints. This software is ALPHA, only use for development, testing, and experimentation. target/release/tursodb --experimental-indexes apinatest_main.db <<<'' 6.81s user 21.18s system 83% cpu 33.643 total ``` PR: 13 seconds ``` jussi@Jussis-MacBook-Pro limbo % time target/release/tursodb --experimental-indexes apinatest_PR.db <<'EOF' create table t(x, y, z unique); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); insert into t select randomblob(1024*128),randomblob(1024*128),randomblob(1024*128) from generate_series(1, 100); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); update t set x = x + 1 WHERE z > randomblob(1024*128); EOF Turso v0.1.3-pre.3 Enter ".help" for usage hints. This software is ALPHA, only use for development, testing, and experimentation. target/release/tursodb --experimental-indexes apinatest_PR.db <<<'' 3.89s user 7.83s system 89% cpu 13.162 total ``` (sqlite: 2 seconds 🤡 ) --- TODO: - [x] Fix whatever issue the simulator caught in CI (#2238 ) - [x] Post a performance comparison - [x] Fix autovacuum test failure - [x] Improve docs - [x] Fix `fill_cell_payload` re-entrancy issue when allocating overflow pages - [x] Add proper PR description Reviewed-by: Pere Diaz Bou <pere-altea@homail.com> Closes #2233
Turso Database
Turso Database is an in-process SQL database, compatible with SQLite.
Features
Turso Database is a work-in-progress, in-process OLTP database engine library written in Rust that has:
- SQLite compatibility for SQL dialect, file formats, and the C API [see document for details]
- Change data capture (CDC) for real-time tracking of database changes.
- Language support for
- Asynchronous I/O support on Linux with
io_uring - OS support for Linux, macOS, and Windows
Roadmap
The following features are on our current roadmap:
BEGIN CONCURRENTfor improved write throughput using multi-version concurrency control (MVCC).- Better schema management, including extended
ALTERsupport, faster schema changes, and strict column types by default. - Incremental computation using DBSP to implement query subscriptions, incremental view maintenance, and triggers.
- Vector indexing for fast approximate vector search, similar to libSQL vector search.
Getting Started
Please see the Turso Database Manual for more information.
💻 Command Line
You can install the latest `turso` release with:
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/tursodatabase/turso/releases/latest/download/turso_cli-installer.sh | sh
Then launch the shell to execute SQL statements:
Turso
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database
turso> CREATE TABLE users (id INT, username TEXT);
turso> INSERT INTO users VALUES (1, 'alice');
turso> INSERT INTO users VALUES (2, 'bob');
turso> SELECT * FROM users;
1|alice
2|bob
You can also build and run the latest development version with:
cargo run
MCP Server Mode
The Turso CLI includes a built-in Model Context Protocol (MCP) server that allows AI assistants to interact with your databases. Start the MCP server with:
tursodb your_database.db --mcp
The MCP server provides seven tools for database interaction:
Available Tools
list_tables- List all tables in the databasedescribe_table- Describe the structure of a specific tableexecute_query- Execute read-only SELECT queriesinsert_data- Insert new data into tablesupdate_data- Update existing data in tablesdelete_data- Delete data from tablesschema_change- Execute schema modification statements (CREATE TABLE, ALTER TABLE, DROP TABLE)
Example Usage
The MCP server runs as a single process that handles multiple JSON-RPC requests over stdin/stdout. Here's how to interact with it:
Example with In-Memory Database
cat << 'EOF' | tursodb --mcp
{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {}, "clientInfo": {"name": "client", "version": "1.0"}}}
{"jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": {"name": "schema_change", "arguments": {"query": "CREATE TABLE users (id INTEGER, name TEXT, email TEXT)"}}}
{"jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": {"name": "list_tables", "arguments": {}}}
{"jsonrpc": "2.0", "id": 4, "method": "tools/call", "params": {"name": "insert_data", "arguments": {"query": "INSERT INTO users VALUES (1, 'Alice', 'alice@example.com')"}}}
{"jsonrpc": "2.0", "id": 5, "method": "tools/call", "params": {"name": "execute_query", "arguments": {"query": "SELECT * FROM users"}}}
EOF
Example with Existing Database
# Working with an existing database file
cat << 'EOF' | tursodb mydb.db --mcp
{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {}, "clientInfo": {"name": "client", "version": "1.0"}}}
{"jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": {"name": "list_tables", "arguments": {}}}
EOF
🦀 Rust
cargo add turso
Example usage:
let db = Builder::new_local("sqlite.db").build().await?;
let conn = db.connect()?;
let res = conn.query("SELECT * FROM users", ()).await?;
✨ JavaScript
npm i @tursodatabase/turso
Example usage:
import { Database } from '@tursodatabase/turso';
const db = new Database('sqlite.db');
const stmt = db.prepare('SELECT * FROM users');
const users = stmt.all();
console.log(users);
🐍 Python
pip install pyturso
Example usage:
import turso
con = turso.connect("sqlite.db")
cur = con.cursor()
res = cur.execute("SELECT * FROM users")
print(res.fetchone())
🦫 Go
- Clone the repository
- Build the library and set your LD_LIBRARY_PATH to include turso's target directory
cargo build --package limbo-go
export LD_LIBRARY_PATH=/path/to/limbo/target/debug:$LD_LIBRARY_PATH
- Use the driver
go get github.com/tursodatabase/turso
go install github.com/tursodatabase/turso
Example usage:
import (
"database/sql"
_ "github.com/tursodatabase/turso"
)
conn, _ = sql.Open("sqlite3", "sqlite.db")
defer conn.Close()
stmt, _ := conn.Prepare("select * from users")
defer stmt.Close()
rows, _ = stmt.Query()
for rows.Next() {
var id int
var username string
_ := rows.Scan(&id, &username)
fmt.Printf("User: ID: %d, Username: %s\n", id, username)
}
☕️ Java
We integrated Turso Database into JDBC. For detailed instructions on how to use Turso Database with java, please refer to the README.md under bindings/java.
Contributing
We'd love to have you contribute to Turso Database! Please check out the contribution guide to get started.
Found a data corruption bug? Get up to $1,000.00
SQLite is loved because it is the most reliable database in the world. The next evolution of SQLite has to match or surpass this level of reliability. Turso is built with Deterministic Simulation Testing from the ground up, and is also tested by Antithesis.
Even during Alpha, if you find a bug that leads to a data corruption and demonstrate how our simulator failed to catch it, you can get up to $1,000.00. As the project matures we will increase the size of the prize, and the scope of the bugs.
More details here.
You can see an example of an awarded case on #2049.
Turso core staff are not eligible.
FAQ
Is Turso Database ready for production use?
Turso Database is currently under heavy development and is not ready for production use.
How is Turso Database different from Turso's libSQL?
Turso Database is a project to build the next evolution of SQLite in Rust, with a strong open contribution focus and features like native async support, vector search, and more. The libSQL project is also an attempt to evolve SQLite in a similar direction, but through a fork rather than a rewrite.
Rewriting SQLite in Rust started as an unassuming experiment, and due to its incredible success, replaces libSQL as our intended direction. At this point, libSQL is production ready, Turso Database is not - although it is evolving rapidly. More details here.
Publications
- Pekka Enberg, Sasu Tarkoma, Jon Crowcroft Ashwin Rao (2024). Serverless Runtime / Database Co-Design With Asynchronous I/O. In EdgeSys ‘24. [PDF]
- Pekka Enberg, Sasu Tarkoma, and Ashwin Rao (2023). Towards Database and Serverless Runtime Co-Design. In CoNEXT-SW ’23. [PDF] [Slides]
License
This project is licensed under the MIT license.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Turso Database by you, shall be licensed as MIT, without any additional terms or conditions.
Partners
Thanks to all the partners of Turso!
Contributors
Thanks to all the contributors to Turso Database!