This PR adds few functions to the `turso-db` in order to simplify
exploration of CDC table. Later we will also add API to work with
changes from the code - but SQL support is also useful.
So, this PR adds 2 functions:
1. `table_columns_json_array('<table-name>')` - returns list of current
table column **names** as a single string in JSON array format
2. `bin_record_json_object('<columns-array>', x'<bin-record>')` -
convert record in the SQLite format to the JSON object with keys from
`columns-array`
So, this functions can be used together to extract changes in human-
readable format:
```sql
turso> PRAGMA unstable_capture_data_changes_conn('full');
turso> CREATE TABLE t(a INTEGER PRIMARY KEY, b);
turso> INSERT INTO t VALUES (1, 2), (3, 4);
turso> UPDATE t SET b = 20 WHERE a = 1;
turso> UPDATE t SET a = 30, b = 40 WHERE a = 3;
turso> DELETE FROM t WHERE a = 1;
turso> SELECT
bin_record_json_object(table_columns_json_array('t'), before) before,
bin_record_json_object(table_columns_json_array('t'), after) after
FROM turso_cdc;
┌─────────────────┬────────────────┐
│ before │ after │
├─────────────────┼────────────────┤
│ │ {"a":1,"b":2} │
├─────────────────┼────────────────┤
│ │ {"a":3,"b":4} │
├─────────────────┼────────────────┤
│ {"a":1,"b":2} │ {"a":1,"b":20} │
├─────────────────┼────────────────┤
│ {"a":3,"b":4} │ │
├─────────────────┼────────────────┤
│ {"a":30,"b":40} │ │
├─────────────────┼────────────────┤
│ {"a":1,"b":20} │ │
└─────────────────┴────────────────┘
```
Initially, I thought to implement single function like
`bin_record_json_object('<table-name', x'<bin-record')` but this design
has certain flaws:
1. In case of schema changes this function can return incorrect result
(imagine that you dropped a column and now JSON from CDC mentions some
random subset of columns). While this feature is unstable - `turso-db`
should avoid silent incorrect behavior at all cost
2. Single-function design provide no way to deal with schema changes
3. The API is unsound and user can think that under the hood `turso-db`
will select proper schema for the record (but this is actually
impossible with current CDC implementation)
So, I decided to stop with two-functions design which cover drawbacks
mentioned above to some extent
1. First concern still remains valid
2. Two-functions design provides a way to deal with schema changes. For
example, user can maintain simple `cdc_schema_changes` table and log
result of `table_columns_json_array` before applying breaking schema
changes.
* Obviously, this is not ideal UX - but it suits my needs: I don't
want to design schema changes capturing, but also I don't want to block
users and provide a way to have a workaround for scenarios which are not
natively supported by CDC
3. Subjectively, I think that API became a bit more clear about the
machinery of these two functions as user see that it extract column list
of the table (without any context) and then feed it to the
`bin_record_json_object` function.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes #2057
Turso Database
Turso Database is an in-process SQL database, compatible with SQLite.
Features and Roadmap
Turso Database is a work-in-progress, in-process OLTP database engine library written in Rust that has:
- SQLite compatibility [doc] for SQL dialect, file formats, and the C API
- Language bindings for JavaScript/WebAssembly, Rust, Go, Python, and Java
- Asynchronous I/O support on Linux with
io_uring - OS support for Linux, macOS, and Windows
In the future, we will be also working on:
BEGIN CONCURRENTfor improved write throughput.- Indexing for vector search.
- Improved schema management including better
ALTERsupport and strict column types by default.
Getting Started
Please see the Turso Database Manual for more information.
💻 Command Line
You can install the latest `turso` release with:
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/tursodatabase/turso/releases/latest/download/turso_cli-installer.sh | sh
Then launch the shell to execute SQL statements:
Turso
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database
turso> CREATE TABLE users (id INT, username TEXT);
turso> INSERT INTO users VALUES (1, 'alice');
turso> INSERT INTO users VALUES (2, 'bob');
turso> SELECT * FROM users;
1|alice
2|bob
You can also build and run the latest development version with:
cargo run
🦀 Rust
cargo add turso
Example usage:
let db = Builder::new_local("sqlite.db").build().await?;
let conn = db.connect()?;
let res = conn.query("SELECT * FROM users", ()).await?;
✨ JavaScript
npm i @tursodatabase/turso
Example usage:
import { Database } from '@tursodatabase/turso';
const db = new Database('sqlite.db');
const stmt = db.prepare('SELECT * FROM users');
const users = stmt.all();
console.log(users);
🐍 Python
pip install pyturso
Example usage:
import turso
con = turso.connect("sqlite.db")
cur = con.cursor()
res = cur.execute("SELECT * FROM users")
print(res.fetchone())
🐹 Go
- Clone the repository
- Build the library and set your LD_LIBRARY_PATH to include turso's target directory
cargo build --package limbo-go
export LD_LIBRARY_PATH=/path/to/limbo/target/debug:$LD_LIBRARY_PATH
- Use the driver
go get github.com/tursodatabase/turso
go install github.com/tursodatabase/turso
Example usage:
import (
"database/sql"
_ "github.com/tursodatabase/turso"
)
conn, _ = sql.Open("sqlite3", "sqlite.db")
defer conn.Close()
stmt, _ := conn.Prepare("select * from users")
defer stmt.Close()
rows, _ = stmt.Query()
for rows.Next() {
var id int
var username string
_ := rows.Scan(&id, &username)
fmt.Printf("User: ID: %d, Username: %s\n", id, username)
}
☕️ Java
We integrated Turso Database into JDBC. For detailed instructions on how to use Turso Database with java, please refer to the README.md under bindings/java.
Contributing
We'd love to have you contribute to Turso Database! Please check out the contribution guide to get started.
FAQ
Is Turso Database ready for production use?
Turso Database is currently under heavy development and is not ready for production use.
How is Turso Database different from Turso's libSQL?
Turso Database is a project to build the next evolution of SQLite in Rust, with a strong open contribution focus and features like native async support, vector search, and more. The libSQL project is also an attempt to evolve SQLite in a similar direction, but through a fork rather than a rewrite.
Rewriting SQLite in Rust started as an unassuming experiment, and due to its incredible success, replaces libSQL as our intended direction. At this point, libSQL is production ready, Turso Database is not - although it is evolving rapidly. More details here.
Publications
- Pekka Enberg, Sasu Tarkoma, Jon Crowcroft Ashwin Rao (2024). Serverless Runtime / Database Co-Design With Asynchronous I/O. In EdgeSys ‘24. [PDF]
- Pekka Enberg, Sasu Tarkoma, and Ashwin Rao (2023). Towards Database and Serverless Runtime Co-Design. In CoNEXT-SW ’23. [PDF] [Slides]
License
This project is licensed under the MIT license.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Turso Database by you, shall be licensed as MIT, without any additional terms or conditions.
Partners
Thanks to all the partners of Turso!
Contributors
Thanks to all the contributors to Turso Database!


