### Async IO performance, part 0 Relatively small and focused PR that mainly does two things, will add a .md document of the proposed/planned improvements to the io_uring module to fully revamp our async IO. 1. **Registration of file descriptors.** At startup, by calling `io_uring_register_files_sparse` we can allocate an array in shared kernel/user space by calling register_files_sparse which initializes each slot to `-1`, and when we open a file we call `io_uring_register_files_update`, providing an index into this array and `fd`. Then for the IO submission, we can reference the index into this array instead of the fd, saving the kernel the work of looking up the fd in the process file table, incrementing the reference count, doing the operation, then finally decrementing the refcount. Instead the kernel can just index into the array and do the operation. This especially provides an improvement for cases like this, where files are open for long periods of time, which the kernel will perform many operations on. The eventual goal of this, is to use Fixed read/write operations, where both the file descriptor and the underlying buffer is registered with the kernel. There is another branch continuing this work, that introduces a buffer pool that memlock's one large 32MB arena mmap and tries to use that wherever possible. These Fixed operations are essentially the "holy grail" of io_uring performance (for file operations). 2. **!Vectored IO** This is kind of backwards, because the goal is to indeed implement proper vectored IO and I'm removing some of the plumbing in this PR, but currently we have been using `Writev`/`Readv`, while never submitting > 1 iovec at a time. Writes to the WAL, especially, would benefit immensely from vectored IO, as it is append-only and therefore all writes are contiguous. Regular checkpointing/cache flushing to disk can also be adapted to aggregate these writes and submit many in a single system call/opcode. Until this is implemented, the bookkeeping and iovecs are unnecessary noise/overhead, so let's temporarily remove them and revert to normal `read`/`write` until they are needed and it can be designed from scratch. 3. **Flags** `setup_single_issuer` hints to the kernel that `IOURING_ENTER` calls will all be sent from a single thread, and `setup_coop_taskrun` removes some unnecessary kernel interrupts for providing cqe's which most single threaded applications do not need. Both these flags demonstrate modest improvement of performance. Closes #2127
Turso Database
Turso Database is an in-process SQL database, compatible with SQLite.
Features and Roadmap
Turso Database is a work-in-progress, in-process OLTP database engine library written in Rust that has:
- SQLite compatibility [doc] for SQL dialect, file formats, and the C API
- Language bindings for JavaScript/WebAssembly, Rust, Go, Python, and Java
- Asynchronous I/O support on Linux with
io_uring - OS support for Linux, macOS, and Windows
In the future, we will be also working on:
BEGIN CONCURRENTfor improved write throughput.- Indexing for vector search.
- Improved schema management including better
ALTERsupport and strict column types by default.
Getting Started
Please see the Turso Database Manual for more information.
💻 Command Line
You can install the latest `turso` release with:
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/tursodatabase/turso/releases/latest/download/turso_cli-installer.sh | sh
Then launch the shell to execute SQL statements:
Turso
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database
turso> CREATE TABLE users (id INT, username TEXT);
turso> INSERT INTO users VALUES (1, 'alice');
turso> INSERT INTO users VALUES (2, 'bob');
turso> SELECT * FROM users;
1|alice
2|bob
You can also build and run the latest development version with:
cargo run
🦀 Rust
cargo add turso
Example usage:
let db = Builder::new_local("sqlite.db").build().await?;
let conn = db.connect()?;
let res = conn.query("SELECT * FROM users", ()).await?;
✨ JavaScript
npm i @tursodatabase/turso
Example usage:
import { Database } from '@tursodatabase/turso';
const db = new Database('sqlite.db');
const stmt = db.prepare('SELECT * FROM users');
const users = stmt.all();
console.log(users);
🐍 Python
pip install pyturso
Example usage:
import turso
con = turso.connect("sqlite.db")
cur = con.cursor()
res = cur.execute("SELECT * FROM users")
print(res.fetchone())
🦫 Go
- Clone the repository
- Build the library and set your LD_LIBRARY_PATH to include turso's target directory
cargo build --package limbo-go
export LD_LIBRARY_PATH=/path/to/limbo/target/debug:$LD_LIBRARY_PATH
- Use the driver
go get github.com/tursodatabase/turso
go install github.com/tursodatabase/turso
Example usage:
import (
"database/sql"
_ "github.com/tursodatabase/turso"
)
conn, _ = sql.Open("sqlite3", "sqlite.db")
defer conn.Close()
stmt, _ := conn.Prepare("select * from users")
defer stmt.Close()
rows, _ = stmt.Query()
for rows.Next() {
var id int
var username string
_ := rows.Scan(&id, &username)
fmt.Printf("User: ID: %d, Username: %s\n", id, username)
}
☕️ Java
We integrated Turso Database into JDBC. For detailed instructions on how to use Turso Database with java, please refer to the README.md under bindings/java.
Contributing
We'd love to have you contribute to Turso Database! Please check out the contribution guide to get started.
FAQ
Is Turso Database ready for production use?
Turso Database is currently under heavy development and is not ready for production use.
How is Turso Database different from Turso's libSQL?
Turso Database is a project to build the next evolution of SQLite in Rust, with a strong open contribution focus and features like native async support, vector search, and more. The libSQL project is also an attempt to evolve SQLite in a similar direction, but through a fork rather than a rewrite.
Rewriting SQLite in Rust started as an unassuming experiment, and due to its incredible success, replaces libSQL as our intended direction. At this point, libSQL is production ready, Turso Database is not - although it is evolving rapidly. More details here.
Publications
- Pekka Enberg, Sasu Tarkoma, Jon Crowcroft Ashwin Rao (2024). Serverless Runtime / Database Co-Design With Asynchronous I/O. In EdgeSys ‘24. [PDF]
- Pekka Enberg, Sasu Tarkoma, and Ashwin Rao (2023). Towards Database and Serverless Runtime Co-Design. In CoNEXT-SW ’23. [PDF] [Slides]
License
This project is licensed under the MIT license.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Turso Database by you, shall be licensed as MIT, without any additional terms or conditions.
Partners
Thanks to all the partners of Turso!
Contributors
Thanks to all the contributors to Turso Database!


