Pekka Enberg 0d17d35ef4 Merge 'Change data capture' from Nikita Sivukhin
This PR add basic CDC functionality to the `turso-db`.
### Feature components
1. `unstable_capture_data_changes_conn` pragma which allow user to turn
on/off CDC logging for **specific connection**
    * CDC will have multiple modes, but for now only `off` / `rowid-
only` are supported
    * Default CDC table is `turso_cdc` but user can override this with
`PRAGMA` update syntax and use arbitrary table for the CDC needs
      * This can be helpful in future if turso will need to break table
format compatibility and custom tables can be a way to migrate between
different schemas
    * Update syntax for the pragma accepts one string argument in
format, where only mode is set or custom cdc table name is provided as
second part of the string, separated with comma from the mode
```sql
turso> PRAGMA unstable_capture_data_changes_conn('rowid-only');
turso> PRAGMA unstable_capture_data_changes_conn('off');
turso> PRAGMA unstable_capture_data_changes_conn('rowid-only,custom_cdc_table');
turso> PRAGMA unstable_capture_data_changes_conn;
┌────────────┬──────────────────┐
│ mode       │ table            │
├────────────┼──────────────────┤
│ rowid-only │ custom_cdc_table │
└────────────┴──────────────────┘
```
2. CDC table schema right now is simple but it will be evolved soon to
support logging of row values before/after the change:
```sql
CREATE TABLE custom_cdc_table (
  operation_id INTEGER PRIMARY KEY AUTOINCREMENT,
  operation_time INTEGER, -- unixepoch() at the moment of insert, can drift if machine clocks is not monotonic
  operation_type INTEGER, -- -1 = delete, 0 = update, 1 = insert
  table_name TEXT,
  id
)
```
  * Note, that `operation_id` is marked as `AUTOINCREMENT` but `turso-
db` needs to implement
https://github.com/tursodatabase/turso/issues/1976 in order to properly
support that keyword
3. Query planner changes are made in `INSERT`/`UPDATE`/`DELETE` plans in
order to emit updates to the CDC table for changes in the table
  * Note, that row `UPDATE` which change primary key generate `DELETE` +
`INSERT` statement instead of single `UPDATE`
### Implementation details
- `PRAGMA` to enable CDC is **unstable** which means that publicly
visible side-effects/public API can change in future (and it will change
soon in order to support more rich CDC modes)
- CDC table is just a regular table with its benefits and downsides:
  * benefits: user can perform maintenance operations with that table
just with regular SQL like `DELETE FROM turso_cdc WHERE operation_id <
?` to cleanup old not needed CDC entries
  * downsides: user can accidentally make unwanted change to CDC table
- Changes to CDC table is not logged to itself
  * Note, that different connections (e.g. `C1`, `C2`) can have
different CDC tables set (e.g. `A` and `B`) - in which case changes made
to CDC table `B` through connection `C1` will be reflected in CDC table
`A`

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1926
2025-07-07 18:03:07 +03:00
2025-07-07 12:38:21 +02:00
2025-06-18 21:26:23 +03:00
2025-07-07 13:24:45 +03:00
2025-06-30 10:29:34 +03:00
2025-06-30 09:54:13 +03:00
2025-03-29 14:46:11 +02:00
2025-07-07 12:50:10 +04:00
2025-01-14 18:37:26 +02:00
2025-06-11 11:32:17 -03:00
2025-04-15 12:45:46 -03:00
2025-07-04 12:53:58 +03:00
2025-07-04 12:54:58 +03:00
2025-06-30 23:57:52 +03:00
2025-05-02 19:26:44 -03:00
2025-07-04 10:13:50 +03:00
2025-01-30 18:24:19 -03:00
2025-06-19 09:41:01 +03:00
2024-07-12 13:07:34 -07:00
2024-07-12 12:38:56 -07:00
2025-07-06 10:37:49 +03:00
2025-06-26 21:05:02 +03:00
2025-07-02 10:57:46 -04:00

Turso Database

Turso Database

Turso Database is an in-process SQL database, compatible with SQLite.

PyPI PyPI PyPI

Chat with the Core Developers on Discord

Chat with other users of Turso (and Turso Cloud) on Discord


Features and Roadmap

Turso Database is a work-in-progress, in-process OLTP database engine library written in Rust that has:

  • SQLite compatibility [doc] for SQL dialect, file formats, and the C API
  • Language bindings for JavaScript/WebAssembly, Rust, Go, Python, and Java
  • Asynchronous I/O support on Linux with io_uring
  • OS support for Linux, macOS, and Windows

In the future, we will be also working on:

  • BEGIN CONCURRENT for improved write throughput.
  • Indexing for vector search.
  • Improved schema management including better ALTER support and strict column types by default.

Getting Started

Please see the Turso Database Manual for more information.

💻 Command Line
You can install the latest `turso` release with:
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/tursodatabase/turso/releases/latest/download/turso_cli-installer.sh | sh

Then launch the shell to execute SQL statements:

Turso
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database
turso> CREATE TABLE users (id INT, username TEXT);
turso> INSERT INTO users VALUES (1, 'alice');
turso> INSERT INTO users VALUES (2, 'bob');
turso> SELECT * FROM users;
1|alice
2|bob

You can also build and run the latest development version with:

cargo run
🦀 Rust
cargo add turso

Example usage:

let db = Builder::new_local("sqlite.db").build().await?;
let conn = db.connect()?;

let res = conn.query("SELECT * FROM users", ()).await?;
JavaScript
npm i @tursodatabase/turso

Example usage:

import { Database } from '@tursodatabase/turso';

const db = new Database('sqlite.db');
const stmt = db.prepare('SELECT * FROM users');
const users = stmt.all();
console.log(users);
🐍 Python
pip install pyturso

Example usage:

import turso

con = turso.connect("sqlite.db")
cur = con.cursor()
res = cur.execute("SELECT * FROM users")
print(res.fetchone())
🐹 Go
  1. Clone the repository
  2. Build the library and set your LD_LIBRARY_PATH to include turso's target directory
cargo build --package limbo-go
export LD_LIBRARY_PATH=/path/to/limbo/target/debug:$LD_LIBRARY_PATH
  1. Use the driver
go get github.com/tursodatabase/turso
go install github.com/tursodatabase/turso

Example usage:

import (
    "database/sql"
    _ "github.com/tursodatabase/turso"
)

conn, _ = sql.Open("sqlite3", "sqlite.db")
defer conn.Close()

stmt, _ := conn.Prepare("select * from users")
defer stmt.Close()

rows, _ = stmt.Query()
for rows.Next() {
    var id int
    var username string
    _ := rows.Scan(&id, &username)
    fmt.Printf("User: ID: %d, Username: %s\n", id, username)
}
Java

We integrated Turso Database into JDBC. For detailed instructions on how to use Turso Database with java, please refer to the README.md under bindings/java.

Contributing

We'd love to have you contribute to Turso Database! Please check out the contribution guide to get started.

FAQ

Is Turso Database ready for production use?

Turso Database is currently under heavy development and is not ready for production use.

How is Turso Database different from Turso's libSQL?

Turso Database is a project to build the next evolution of SQLite in Rust, with a strong open contribution focus and features like native async support, vector search, and more. The libSQL project is also an attempt to evolve SQLite in a similar direction, but through a fork rather than a rewrite.

Rewriting SQLite in Rust started as an unassuming experiment, and due to its incredible success, replaces libSQL as our intended direction. At this point, libSQL is production ready, Turso Database is not - although it is evolving rapidly. More details here.

Publications

  • Pekka Enberg, Sasu Tarkoma, Jon Crowcroft Ashwin Rao (2024). Serverless Runtime / Database Co-Design With Asynchronous I/O. In EdgeSys 24. [PDF]
  • Pekka Enberg, Sasu Tarkoma, and Ashwin Rao (2023). Towards Database and Serverless Runtime Co-Design. In CoNEXT-SW 23. [PDF] [Slides]

License

This project is licensed under the MIT license.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Turso Database by you, shall be licensed as MIT, without any additional terms or conditions.

Partners

Thanks to all the partners of Turso!

Contributors

Thanks to all the contributors to Turso Database!

Description
No description provided
Readme 43 MiB
Languages
Rust 76.8%
Tcl 6.6%
C 6.4%
Dart 2.4%
Java 2.3%
Other 5.3%