This PR emit CDC entries as changes in `sqlite_schema` table for DDL
statements: `CREATE TABLE` / `CREATE INDEX` / etc.
The logic is a bit tricky as under the hood `turso` can do some implicit
DDL operations like:
1. Creating auto-indexes in case of `CREATE TABLE`
2. Deletion of all attached indices in case of `DROP TABLE`
```
turso> PRAGMA unstable_capture_data_changes_conn('full');
turso> CREATE TABLE t(x, y, z UNIQUE, q, PRIMARY KEY (x, y));
turso> CREATE INDEX t_xy ON t(x, y);
turso> CREATE TABLE q(a, b, c);
turso> ALTER TABLE q DROP COLUMN b;
turso> SELECT
change_id,
id,
change_type,
table_name,
bin_record_json_object(table_columns_json_array(table_name), before) AS before,
bin_record_json_object(table_columns_json_array(table_name), after) AS after
FROM turso_cdc;
┌───────────┬────┬─────────────┬───────────────┬─────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ change_id │ id │ change_type │ table_name │ before │ after │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 1 │ 2 │ 1 │ sqlite_schema │ │ {"type":"table","name":"t","tbl_name":"t","rootpage":3,"sql":"CREA… │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 2 │ 5 │ 1 │ sqlite_schema │ │ {"type":"index","name":"t_xy","tbl_name":"t","rootpage":6,"sql":"C… │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 3 │ 6 │ 1 │ sqlite_schema │ │ {"type":"table","name":"q","tbl_name":"q","rootpage":7,"sql":"CREA… │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 4 │ 6 │ 0 │ sqlite_schema │ {"type":"table","name":"q","tbl_name":"q","rootpage":7,"sql":"CREA… │ {"type":"table","name":"q","tbl_name":"q","rootpage":7,"sql":"CREA… │
└───────────┴────┴─────────────┴───────────────┴─────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘
```
For now, CDC capture only all explicit operations and ignore all
implicit operations. The reasoning for that is that one use case for CDC
is to apply logical changes as is with simple SQL statements - but if
implicit operations will be logged to the CDC table too - we can have
hard times using simple SQL statement (for example, creation of
`autoindices` will always work; implicit deletion of indices for `DROP
TABLE` also can lead to some troubles and force us to is `DROP INDEX IF
EXISTS ...` statements + we will need to filter out autoindices in this
case too).
Also, to simplify PR, for now `DatabaseTape` from `turso-sync` package
just ignore all schema changes from CDC table.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2426
The built-in `unreachable!` macro, believe it or not is just an alias
for `panic!` and does not actually provide the compiler with a hint that
the path is not reachable.
This provides a wrapper around the actual
`std::hint::unreachable_unchecked()`, to be used only in the very hot
path of `execute` where it is not possible to be the incorrect variant.
Closes#2459
## Beef
Removes cloning of `WriteState` in `balance_non_root()` and removes the
`Clone` implementation from `WriteState`. Not only is it unnecessary to
clone the state, but even having the `Clone` implementation available is
a real regression risk in `insert_into_page()` as it will deep clone
vectors that we require stable raw pointers to in `fill_cell_payload()`
## Details
In removing the cloning of `WriteState` in `balance_non_root()`, this
replaces the `next_write_state` logic with a plain loop, which ends up
looking like a lot of changes, but it's really not. Main things that
were changed:
- Loop and mutate state instead of create new state and return
- Make `WriteInfo` an `Option` in `WriteState` for ergonomics so it can
be `.take()`:n instead of `.clone()`:d
- Clone a `Rc<Pager>` in `balance_non_root()` so that we can call
`pager.do_allocate_page()` directly instead of `self.allocate_page()`
which is just a facade for the pager method, and would violate borrowing
rules.
- assign `let usable_space = self.usable_space()` at the beginning of
the balance loop for similar reasons.
- Make the balance debug validation methods static so they don't require
a reference to self, for similar reasons
Reviewed-by: Nikita Sivukhin (@sivukhin)
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#2466
## Problem
We currently clone `WriteState` in every loop iteration of
`insert_into_page()`, which was probably done for borrow checker
reasons, but since `WriteState` has expanded to include buffers that
must not be moved in memory or dropped, it has necessitated a really
annoying workaround of wrapping the buffers in `Arc<Mutex>>` which is
just completely wasteful.
## Fix
Do not clone `WriteState` in `insert_into_page()`, and instead work with
the borrow checker a bit more. Note that `WriteState` still _implements_
`Clone` because it's also cloned in `balance_non_root()` - that can be a
separate refactor.
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2464
## Background
We currently clone `WriteState` on every iteration of
`insert_into_page()`,
presumably for _Borrow Checker Reasons (tm)_.
## Problem
There was a bug in `WriteState::Insert` handling where if
`fill_cell_payload()`
returned IO, the `fill_cell_payload_state` was not updated in
`write_info.state`, leading to an infinite loop of allocating new pages.
Further, the `new_payload` vector was also always deep-cloned.
## Fix
Update the state if `fill_cell_payload()` returns IO. Because of
`WriteState` cloning we also need to `Arc<Mutex>` wrap the vector so
that the underlying data buffer doesn't get cloned the next time
`WriteState` gets cloned, since `fill_cell_payload` relies on raw
pointers to work. Left a couple of prominent FIXMEs about this.
## Notes
This bug was surfaced by, but not caused by,
https://github.com/tursodatabase/turso/pull/2400.
Closes#2463
We currently clone WriteState on every iteration of `insert_into_page()`,
presumably for Borrow Checker Reasons (tm).
There was a bug in `WriteState::Insert` handling where if `fill_cell_payload()`
returned IO, the `fill_cell_payload_state` was not updated in
`write_info.state`, leading to an infinite loop of allocating new pages.
This bug was surfaced by, but not caused by, #2400.
We are doing unsafe mut borrowing in `PageContent` currently, and when
new BufferPool is merged, it will be all pointers into an arena anyway.
We just need to be sure to clone the Arc and reference the buffers in
the completion callbacks so they are ensured to live for async IO.
naturally in true spirit of all my previous PR's, I needed to introduce
one while another is open that causes an absurd amount of conflicts.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2456
After #2258, we no longer need to clone/deep copy `PageContent` or
`Buffer`.
We can remove the implementation to make any copying of page data
explicit.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2453
SQLite generates those in aggregations like min / max with collation
information either in the table definition or in the column expression.
We currently generate the wrong result here, and properly generating the
bytecode instruction fixes it.
Closes#2440
## Fix 1
Do not start a read transaction when a SELECT is not going to access the
database, which means we can avoid checking whether the schema has
changed.
## Fix 2
Add a field `accesses_db` to `Program` and `Statement` so we can avoid
even checking for `SchemaUpdated` errors when it's not possible to get
one.
## Fix 3
Avoid doing any work in `commit_txn` when not in a transaction. This
optimization is only enabled when `mv_store.is_none()`, because MVCC has
its own logic and this doesn't work with MVCC enabled, and honestly I'm
too tired to find out why. Left an inline comment about it, though.
```sql
Execute `SELECT 1`/limbo_execute_select_1
time: [21.440 ns 21.513 ns 21.586 ns]
change: [-60.766% -60.616% -60.453%] (p = 0.00 < 0.05)
Performance has improved.
```
Effect is even more dramatic in CI where the latency is down over 80%
Closes#2441
In preparation for tracking IO Completions, we need to start to return
IO in places where completions are created. Doing some more plumbing now
to avoid bigger PRs for the future
Closes#2438
## Beef
Adds `AddColumn`, `DropColumn`, `RenameColumn`
## Details
- Previously the test was hardcoded to assume there's always 2 named
columns, so changed a bunch of things for this reason
- Still assumes the primary key column is always `id` and is never
renamed or dropped etc.
Closes#2434
