Closes#2241
## What
When an index interior cell is deleted, it steals the leaf cell with the
largest key in its left subtree, deletes the old interior cell and then
replaces it with the stolen cell. This ensures the binary-search-tree
aspect of the btree remains correct. However, this can cause a situation
where both are true:
1. The leaf page is now UNDERFULL and must be rebalanced
2. The leaf's IMMEDIATE parent page is now OVERFULL and must be
rebalanced
## Why is this a problem
We simply didn't support the case where:
- Leaf page P is unbalanced and rebalancing starts on it
- Its immediate parent is ALSO unbalanced and _overflows_.
We had an assertion against this happening (see #2241)
## The fix
Allow exactly 1 overflow cell in the parent under very particular
conditions:
1. The parent page must be an index interior page
2. The parent must be positioned exactly at the divider cell whose left
child page underflows
This is the _only_ case where the immediate parent of a page about to
undergo rebalancing can have overflow cells.
## Implementation details
The parent overflow cell is folded into `cell_array` fairly early on and
`parent.overflow_cells` is cleared. However we need to be careful with
`cell_idx` for dividers other than the overflow cell because they get
shifted left on the page in `drop_cell()`. I've added a long comment
about this.
## Testing
Adds fuzz test that does inserts and deletes on an index btree and
asserts that all the expected keys are found at the end in the right
order. This test runs into this case quite frequently so I was able to
verify it.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2243
Let's make sure we don't end up in a weird situation by appending frames
one by one and we can later think of optimizations.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2034
SQLite behavior is: if another connection has modified the DB when a
read tx starts, it must clear its page cache due to the potentiality
of there being stale versions of pages in it.
In the future, we may want to do either:
1. a more granular invalidation logic for per-conn cache, or
2. a shared versioned page cache
But right now we must follow SQLite to make our current behavior not
corrupt data
This PR partially fixes issue when schema changes were invisible after
WAL sync calls. Now, `wal_insert_end` always read fresh schema cookie
and re-parse schema from scratch if cookie changed.
Generally, the problem of "silent" schema update can be more generic
if(when?) `turso-db` will support multi-process setup. But for now only
single-process can work with `turso-db`, so I decided to inject re-parse
logic explicitly in WAL raw API in order to not introduce any
unnecessary overhead in the ordinary execution path.
This fix is not complete, as if we will have already prepared statements
- they should be re-prepared too in case of schema changes. But this
problem already tracked in the PR
https://github.com/tursodatabase/turso/pull/2214
Reviewed-by: Pedro Muniz (@pedrocarlo)
Closes#2246
WAL insert API introduced in the #2231 works incorrectly as it never
mark inserted pages as dirty.
This PR fixes this issue and also add simple fuzz test which fails
without fixes.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2245
Make `add_dirty` helper to set flag and add page to the dirt list. This
makes API safer as now its harder to do one thing and forget about
another (which can lead to DB corruption).
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2244
The SetCookie opcode is used, among other things, to notify the
transaction of schema changes. We are not issuing it on DropTable.
Without it, the transaction thinks the schema hasn't changed, and does
not update the schema of the connection back to the database.
SQLite will, of course, issue it:
35 DropTable 0 0 0 foo 0
36 SetCookie 0 1 2 0
Unfortunately I don't have a unit test that breaks with this, because
the one that is supposed to break is having, let's put it this way,
bigger problems.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2249
This PR adds a const associated value on the VTabModule trait,
`READONLY` defaulted to `true`, so we can bail early when a write
operation is done on an invalid vtable.
This prevents extensions from having to implement `insert`,`update`,
`delete` just to return `Error::ReadOnly`, and prevents us from having
to step through `VUpdate` just to error out.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2247
This PR adds support for `INSERT` queries with explicit value for
`rowid` column (not thought rowid alias):
```
turso> create table t(x, y, z);
turso> insert into t(rowid, x, y, z) values (10, 1, 2, 3);
turso> select rowid, * from t;
┌───────┬───┬───┬───┐
│ rowid │ x │ y │ z │
├───────┼───┼───┼───┤
│ 10 │ 1 │ 2 │ 3 │
└───────┴───┴───┴───┘
```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2239
Before this update, the entire immutable record was **fully**
deserialized **every** time it was compared in the sorter.
This PR extends the sorter with incremental deserialization of record
keys, only when needed and only if they weren’t already deserialized in
a previous iteration.
I hate that we panic on failed deserialization in `cmp`, but
unfortunately, I can’t return `Result` as part of this interface.
Looking for feedback around a better way to handle this.
Alternatively, I could store the deserialization error as part of
`SortableImmutableRecord` and check it before returning the record in
`next`, thereby deferring the error handling. The downside of this
approach is that it complicates debugging, since the error will be
completely decoupled from the place where it occurs.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2207
# Fix SUM aggregate function for mixed types
Fixes#2133
The SUM aggregate function was returning incorrect results when
processing tables with mixed numeric and non-numeric values. According
to SQLite documentation:
> "If any input to sum() is neither an integer nor a NULL, then sum()
returns a floating point value"
[*](https://sqlite.org/lang_aggfunc.html)
Now both SQLite and Turso yield the same output of 44.0.
--
I modified `Sum` to increment only for numeric values, skipping non-
numeric values. However, if we have mixed numeric values or non-numeric
values, we return a float output. Added a flag to keep track of it.
as pointed out by @FHaggs , If there are no non-NULL input rows then
sum() returns NULL but total() returns 0.0. I decided to include it in
this PR as well. Empty was such a natural test case.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2182
The SetCookie opcode is used, among other things, to notify the
transaction of schema changes. We are not issuing it on DropTable.
Without it, the transaction thinks the schema hasn't changed, and does
not update the schema of the connection back to the database.
SQLite will, of course, issue it:
35 DropTable 0 0 0 foo 0
36 SetCookie 0 1 2 0
Unfortunately I don't have a unit test that breaks with this, because
the one that is supposed to break is having, let's put it this way,
bigger problems.
This PR implements missing raw WAL API from LibSQL for future use for
offline-sync feature:
1. `wal_insert_begin` - begin WAL session by opening WAL read/write
transaction
2. `wal_insert_end` - finish WAL session by closing WAL transaction
opened by `wal_insert_begin` call
3. `wal_insert_frame` - insert frame `frame_no` with raw content `frame`
(WAL frame included)
For now any schema changes will not be reflected after
`wal_insert_frame` because `turso-db` do not re-parse schema without
need. I will fix this in follow up PR.
Reviewed-by: Pekka Enberg <penberg@iki.fi>
Closes#2231
## Background
The `balance_non_root` procedure can end up freeing a page if the pages
to be balanced can fit the required combined number of cells in less
pages, even if the page that triggered balancing is overfull. This can
then free the originally overfull pages, leaving a non-zero
`overflow_cells` on the in-mem representation of the page.
```rust
balance_non_root: page=305, overflow_cells=0
balance_non_root: page=304, overflow_cells=0
balance_non_root: page=302, overflow_cells=1
pre_edit_page(page=304, page_idx=0, new_cells=4, old_cells=1, cells_per_page_old=[1, 3, 9, 0, 0], cells_per_page_new=[4, 9, 9, 0, 0], cell_array_count=9)
edit_page start_old_cells=0 start_new_cells=0 number_new_cells=4 cell_array=9 end_old_cells=1 end_new_cells=4
pre_edit_page(page=305, page_idx=1, new_cells=4, old_cells=1, cells_per_page_old=[1, 3, 9, 0, 0], cells_per_page_new=[4, 9, 9, 0, 0], cell_array_count=9)
edit_page start_old_cells=2 start_new_cells=5 number_new_cells=4 cell_array=9 end_old_cells=3 end_new_cells=9
balance_non_root: sibling_count_new=2, sibling_count=3
// Custom assertion to demonstrate this:
thread 'main' panicked at core/storage/pager.rs:1127:29:
Pager::free_page: In memory page with id 302 has overflow cells
```
## Why is this a problem
Right now this is not an immediate problem, because we always allocate
brand new pages. However, in #2233 we begin to reuse pages from the
freelist for page allocation to improve performance and reduce database
size bloat. In that PR, the `balance_non_root` procedure will calculate
cell counts incorrectly in `edit_page()` and panic if: 1. a new
allocated page is taken from the freelist, 2. the page is still in
memory, and 3. and it still contains `overflow_cells`.
## Solution
Clear `page_contents.overflow_cells` when an in-memory page is freed.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2238
