This is just the bare minimum that I needed to convince myself that this
approach will work. The only views that we support are slices of the
main table: no aggregations, no joins, no projections.
drop view is implemented.
view population is implemented.
deletes, inserts and updates are implemented.
much like indexes before, a flag must be passed to enable views.
When building views (soon), it will be important to know which table
is being deleted. Getting from the cursor id is very cumbersome.
What we are doing here is symmetrical to op_insert, and sqlite also
passes table information in one of the registers (p4)
This fix ensures that `WHERE` conditions are emitted after the `LEFT
JOIN` match flag is set, so rows from the right table are properly
filtered, even when they are `NULL` due to the outer join.
Previously, the query below would return rows where `products.price` was
`NULL`:
```sql
SELECT users.id, price
FROM users
LEFT JOIN products ON users.id = products.id
WHERE products.price IS NOT NULL;
```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2501
This PR emit CDC entries as changes in `sqlite_schema` table for DDL
statements: `CREATE TABLE` / `CREATE INDEX` / etc.
The logic is a bit tricky as under the hood `turso` can do some implicit
DDL operations like:
1. Creating auto-indexes in case of `CREATE TABLE`
2. Deletion of all attached indices in case of `DROP TABLE`
```
turso> PRAGMA unstable_capture_data_changes_conn('full');
turso> CREATE TABLE t(x, y, z UNIQUE, q, PRIMARY KEY (x, y));
turso> CREATE INDEX t_xy ON t(x, y);
turso> CREATE TABLE q(a, b, c);
turso> ALTER TABLE q DROP COLUMN b;
turso> SELECT
change_id,
id,
change_type,
table_name,
bin_record_json_object(table_columns_json_array(table_name), before) AS before,
bin_record_json_object(table_columns_json_array(table_name), after) AS after
FROM turso_cdc;
┌───────────┬────┬─────────────┬───────────────┬─────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ change_id │ id │ change_type │ table_name │ before │ after │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 1 │ 2 │ 1 │ sqlite_schema │ │ {"type":"table","name":"t","tbl_name":"t","rootpage":3,"sql":"CREA… │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 2 │ 5 │ 1 │ sqlite_schema │ │ {"type":"index","name":"t_xy","tbl_name":"t","rootpage":6,"sql":"C… │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 3 │ 6 │ 1 │ sqlite_schema │ │ {"type":"table","name":"q","tbl_name":"q","rootpage":7,"sql":"CREA… │
├───────────┼────┼─────────────┼───────────────┼─────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ 4 │ 6 │ 0 │ sqlite_schema │ {"type":"table","name":"q","tbl_name":"q","rootpage":7,"sql":"CREA… │ {"type":"table","name":"q","tbl_name":"q","rootpage":7,"sql":"CREA… │
└───────────┴────┴─────────────┴───────────────┴─────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘
```
For now, CDC capture only all explicit operations and ignore all
implicit operations. The reasoning for that is that one use case for CDC
is to apply logical changes as is with simple SQL statements - but if
implicit operations will be logged to the CDC table too - we can have
hard times using simple SQL statement (for example, creation of
`autoindices` will always work; implicit deletion of indices for `DROP
TABLE` also can lead to some troubles and force us to is `DROP INDEX IF
EXISTS ...` statements + we will need to filter out autoindices in this
case too).
Also, to simplify PR, for now `DatabaseTape` from `turso-sync` package
just ignore all schema changes from CDC table.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2426
SQLite generates those in aggregations like min / max with collation
information either in the table definition or in the column expression.
We currently generate the wrong result here, and properly generating the
bytecode instruction fixes it.
This PR integrates virtual tables into the query optimizer. It is a
follow-up to https://github.com/tursodatabase/turso/pull/1727.
The most immediate improvement is better support for inner joins
involving TVFs, particularly when TVF arguments are column references.
### Example
The following two queries are semantically equivalent, but require
different join orders to be valid:
```sql
-- TVF depends on `t.id`, so `t` must be evaluated in outer loop
SELECT t.id, series.value
FROM target t, generate_series(t.id, 3) series;
-- Equivalent query, but with reversed table order in the FROM clause
SELECT t.id, series.value
FROM generate_series(t.id, 3) series, target t;
```
Without optimizer integration, the second query would fail because the
planner would attempt to evaluate `generate_series` before `t`. With
this change, the optimizer detects column dependencies and produces the
correct join order in both cases.
### TODO
Support for outer joins with TVFs is still missing and will be addressed
in a follow-up PR.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2439
The side of the binary expression no longer needs to be stored in
`ConstraintInfo`, since the optimizer now guarantees that it is always
on the right. As a result, only the index of the corresponding constraint
needs to be preserved.
This change extends table-valued function support by allowing arguments
to be column references, not only literals.
Virtual tables can now reject a plan by returning an error from
best_index (e.g., when a TVF argument references a table that appears
later in the join order). The planner using this information excludes
invalid plans during join order search.
This change connects virtual tables with the query optimizer.
The optimizer now considers virtual tables during join order search
and invokes their best_index callbacks to determine feasible access
paths.
Currently, this is not a visible change, since none of the existing
extensions return information indicating that a plan is invalid.
Replace panics with proper errors when a valid plan does not exist.
Currently, this never happens because a naive plan is always available.
However, once virtual tables are integrated into the planner, it may
occur—for example, when table-valued function arguments are column
references, and the function cannot be placed in the join order so that
its arguments can be evaluated.
Although this change is effectively a no-op for now, it is extracted
into a separate commit to avoid polluting the one that introduces
virtual table integration with the planner.
Different scan parameters are required for different table types.
Currently, index and iteration direction are only used by B-tree tables,
while the remaining table types don’t require any parameters. Planning
access to virtual tables, however, will require passing additional
information from the planner, such as the virtual table index (distinct
from a B-tree index) and the constraints that must be forwarded to the
`filter` method.
Previously, AccessMethod stored fields like `iter_dir`, `index`, and
`constraint_refs` directly, but these only applied to BTree tables.
Other table types (virtual tables, subqueries) either ignored these
fields or required different parameters entirely.
This change prepares the planner to handle virtual table access methods
with their own specialized parameters.
The `best_index` implementation now returns a ResultCode along with the
IndexInfo. This allows it to signal specific outcomes, such as errors or
constraint violations. This change aligns better with SQLite’s xBestIndex
contract, where cases like missing constraints or invalid combinations of
constraints must not result in a valid plan.
Previously, there were two ways to indicate that a constraint should not
be passed to the filter function: setting `argv_index` to `None` or to
a value less than 1. This was redundant, so now only `None` is used.
Additional changes:
- Update IndexInfo documentation to clarify that constraint_usages must
have exact 1:1 correspondence with input ConstraintInfo array. The code
translating constraints into VFilter arguments heavily relies on this.
- Fix best_index implementation in test extension to comply with new
validation requirements by returning usage entry for each constraint
Resolves#2378.
```
`ALTER TABLE _ RENAME TO _`/limbo_rename_table/
time: [15.645 ms 15.741 ms 15.850 ms]
Found 12 outliers among 100 measurements (12.00%)
8 (8.00%) high mild
4 (4.00%) high severe
`ALTER TABLE _ RENAME TO _`/sqlite_rename_table/
time: [34.728 ms 35.260 ms 35.955 ms]
Found 15 outliers among 100 measurements (15.00%)
8 (8.00%) high mild
7 (7.00%) high severe
```
<img width="1000" height="199" alt="image" src="https://github.com/user-
attachments/assets/ad943355-b57d-43d9-8a84-850461b8af41" />
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2399
