This PR fixes few issues in turso-sync-engine implementation:
1. One step of `pull` implementation works like this:
a. Start write WAL session
b. Revert local changes in WAL
c. Replay WAL frames from remote DB
d. Replay WAL frames produced by local changes applied to the remote
DB copy (`synced`)
My initial thinking was that by executing step (d) we will get the same
schema as before (with same schema cookie) and everything will be fine.
With more deep thinking we can see that it's not fine - as after step
(d) tables created locally can change their root pages (if they were
created locally, for example) - and DB will have "broken" schema
2. Sync engine execute few SQL statements and do not run them to
completion - which basically created "orphaned" locks
In order to fix (1) I decided to introduce another `conn_raw_api`
extension which allows to read and write schema cookie directly in the
transaction. So, the process described above adds step (e) which set
schema cookie to the value strictly greater than the value before.
In order to fix (2) I just fixed all places where statement were dropped
before running to completion.
These fixes are merged together because I explored them by fixing one
new test: `test_sync_single_db_many_pulls_big_payloads`
Closes#2561
Add support for schema changes and granular updates in the
`DatabaseTape` and turso-sync-engine
Now, schema changes made locally will be replicated to the remote too.
Also, `UPDATE`s made locally will touch only changed columns (before we
did `DELETE` + `INSERT` which can overwrite non-conflicting changes from
another device to the same row).
Note, that schema changes replication for now can be pretty dangerous,
as we can't extract proper schema at some moment in time from turso_cdc
and always use latest schema columns. This means that it's better to
avoid `ALTER TABLE ...` to be executed locally, but basic DDL like
`CREATE TABLE / CREATE INDEX / DROP TABLE / DROP INDEX` will work fine
(as columns only appear/disappear for schema in this case).
Closes#2540
- try_wal_watermark_read_page - try to read page from the DB with given WAL watermark value
- wal_changed_pages_after - return set of unique pages changed after watermark WAL position
This PR introduces initial protocol for bidirectional sync with conflict
resolution.
The main addition to the usual `Database` interface are two methods:
1. `push` - push all local changes to the remote. Note, that new changes
from the remote will not be visible during request execution after this
procedure.
2. `pull` - pull all remote changes and apply them locally. Note, that
this procedure require temporary block of writes to the local DB as
internally we will manipulate with opened connections and juggle with
few things under the hood.
## Limitations
* Current implementation exposes only query methods on top of the
**database** - because more careful orchestration will be needed when we
will expose `Connection` and `Statement`
* Schema changes are possible to make through synced Database - but they
are actually not synced to the remote
* Current implementation will amplify storage use by 2x
## Implementation overview
Current approach uses pretty stupid idea to maintain 2 copies of the
database and WAL files:
1. `Draft` - this copy will hold local changes and accept all writes
made to the database
2. `Synced` - this copy will hold DB file and WAL synced with remote
This obviously lead to 2x space amplification, but allow us to implement
sync with conflict resolution without changing `turso-core`.
Under the hood, implementation of main operations looks like this:
1. `push`:
a. Pull all recent changes from the remote to `Synced` DB
b. Transfer local changes from `Draft` to `Synced` with the help of
CDC table
c. Push new WAL frames from `Synced` DB to remote
2. `pull`:
a. Pull all recent changes from the remote to `Synced` DB
b. Transfer local changes from `Draft` to `Synced` with the help of
CDC table
c. Copy `Synced` files (DB and WAL) to the `Draft`
d. Reset `Synced` WAL in order to remove frames made by local changes
from it
As operation 2.c can't be made safely without extra work - `turso-sync`
package internally maintains `active` database which can be either
`Draft` or `Synced` and switch will happen exactly before/after step 2.c
as we will need to move all requests from `Draft` DB to `Synced` due to
explicit copy which we will need to perform.
This switch between Databases creates additional troubles and that's why
in this PR only `Database::query` and `Database::execute` methods are
exposed without prepared statements.
<img width="2062" height="977" alt="Untitled-2025-07-14-1259"
src="https://github.com/user-
attachments/assets/64eb5046-d7cb-4af2-87a0-810c0db7eeb5" />
<img width="2062" height="977" alt="Untitled-2025-07-14-1259(1)"
src="https://github.com/user-
attachments/assets/5c20360c-41db-4100-b0ff-9e47c2682e56" />
Closes#2334
