This PR implements missing raw WAL API from LibSQL for future use for
offline-sync feature:
1. `wal_insert_begin` - begin WAL session by opening WAL read/write
transaction
2. `wal_insert_end` - finish WAL session by closing WAL transaction
opened by `wal_insert_begin` call
3. `wal_insert_frame` - insert frame `frame_no` with raw content `frame`
(WAL frame included)
For now any schema changes will not be reflected after
`wal_insert_frame` because `turso-db` do not re-parse schema without
need. I will fix this in follow up PR.
Reviewed-by: Pekka Enberg <penberg@iki.fi>
Closes#2231
## Background
The `balance_non_root` procedure can end up freeing a page if the pages
to be balanced can fit the required combined number of cells in less
pages, even if the page that triggered balancing is overfull. This can
then free the originally overfull pages, leaving a non-zero
`overflow_cells` on the in-mem representation of the page.
```rust
balance_non_root: page=305, overflow_cells=0
balance_non_root: page=304, overflow_cells=0
balance_non_root: page=302, overflow_cells=1
pre_edit_page(page=304, page_idx=0, new_cells=4, old_cells=1, cells_per_page_old=[1, 3, 9, 0, 0], cells_per_page_new=[4, 9, 9, 0, 0], cell_array_count=9)
edit_page start_old_cells=0 start_new_cells=0 number_new_cells=4 cell_array=9 end_old_cells=1 end_new_cells=4
pre_edit_page(page=305, page_idx=1, new_cells=4, old_cells=1, cells_per_page_old=[1, 3, 9, 0, 0], cells_per_page_new=[4, 9, 9, 0, 0], cell_array_count=9)
edit_page start_old_cells=2 start_new_cells=5 number_new_cells=4 cell_array=9 end_old_cells=3 end_new_cells=9
balance_non_root: sibling_count_new=2, sibling_count=3
// Custom assertion to demonstrate this:
thread 'main' panicked at core/storage/pager.rs:1127:29:
Pager::free_page: In memory page with id 302 has overflow cells
```
## Why is this a problem
Right now this is not an immediate problem, because we always allocate
brand new pages. However, in #2233 we begin to reuse pages from the
freelist for page allocation to improve performance and reduce database
size bloat. In that PR, the `balance_non_root` procedure will calculate
cell counts incorrectly in `edit_page()` and panic if: 1. a new
allocated page is taken from the freelist, 2. the page is still in
memory, and 3. and it still contains `overflow_cells`.
## Solution
Clear `page_contents.overflow_cells` when an in-memory page is freed.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2238
When compiling with features disabled, there are lots of clippy
warnings. This PR silences them.
For the utils file, I am using a bit of a hammer and just allowing
unused stuff in the whole file. Due to the box of utilities nature of
this file, it'll always be the case that things will be unused depending
on the feature-set.
There's no such thing as a read-only connection.
In a normal connection, you can have many attached databases. Some r/o,
some r/w.
To properly fix that, we also need to fix the OpenWrite opcode. Right
now we are passing a name, which is the name of the table. That
parameter is not used anywhere. That is also not what the SQLite opcode
specifies. Same as OpenRead, the p3 register should be the database
index.
With that change, we can - for now - pass the index 0, which is all we
support anyway, and then use that to test if we are r/o.
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#2232
There's no such thing as a read-only connection.
In a normal connection, you can have many attached databases. Some
r/o, some r/w.
To properly fix that, we also need to fix the OpenWrite opcode. Right
now we are passing a name, which is the name of the table. That
parameter is not used anywhere. That is also not what the SQLite opcode
specifies. Same as OpenRead, the p3 register should be the database
index.
With that change, we can - for now - pass the index 0, which is all
we support anyway, and then use that to test if we are r/o.
This PR fixes `wal_read_frame_raw` API
Before, implementation of raw read API read only page content - which is
not enough as we also need page_no and size_after fields from the
header. This PR fixes that and also make few adjustments in the
signatures.
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#2229
`BTreeTable::to_sql` makes us incompatible with SQLite by losing e.g. the original whitespace provided during the CREATE TABLE command.
For now let's fix our tests by regex-replacing every CREATE TABLE in
the entire repo to have exactly 1 space after the table name in the
CREATE TABLE statement.
A write txn can only start if the current snapshot held by writer is
consistent with the one in shared state
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2204
- When an interior index cell is replaced, it can cause the page where
the
replacement happens to overflow OR underflow. On `main` we did not check
this case, because
the interior cell replacement always moves the cursor to a leaf, and if
the leaf
doesn't underflow, then no further balancing happens.
- The solution is to ALWAYS check whether the interior page where the
replacement
happens is underflowing OR overflowing, and balance that page regardless
of whether
the leaf page where the replacement was taken underflows or not.
So summary:
- InteriorCellReplacement: cell deleted from Interior page I,
replacement cell taken from Leaf L
and inserted back to Interior page I.
- If Leaf L underflows:
* balance it first
* then balance I if it overflows OR underflows
- If Leaf L does NOT underflow:
* balance I anyway if it overflows OR underflows
Closes https://github.com/tursodatabase/turso/issues/1701
Closes https://github.com/tursodatabase/turso/issues/2167
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2168
Using `unwrap_or_default` can make `page_size` become 0 in this case,
which can lead to subtracting with overflow in `payload_threshold_max`
in case we have some sort of error. Better to unwrap the error here, as
in release mode we may not have overflow checks enabled to catch this.
Closes#2145
Currently, each record header is decoded at least twice: once to
determine the record size within the read buffer (in order to construct
the `ImmutableRecord` instance), and again later when decoding the
record for comparison. This redundant decoding can have a noticeable
negative impact on performance when records are wide (eg. contain
multiple columns).
This update modifies the (de)serialization format for sorted chunk files
by prepending a record size varint to each record payload. As a result,
only a single varint needs to be decoded to determine the record size,
eliminating the need to decode the full record header during reads.
Closes#2176
- When an interior index cell is replaced, it can cause the page where the
replacement happens to overflow. On `main` we did not check this case, because
the interior cell replacement always moves the cursor to a leaf, and if the leaf
doesn't underflow, then no further balancing happens.
- The solution is to ALWAYS check whether the interior page where the replacement
happens is underflowing OR overflowing, and balance that page regardless of whether
the leaf page where the replacement was taken underflows or not.
So summary:
- InteriorCellReplacement: cell deleted from Interior page I, replacement cell taken from Leaf L
and inserted back to Interior page I.
- If Leaf L underflows:
* balance it first
* then balance I if it overflows OR underflows
- If Leaf L does NOT underflow:
* balance I anyway
Closes#1701Closes#2167
Two of the opcodes we implement (OpenRead and Transaction) should have
an opcode specifying the database to use, but they don't.
Add it, and for now always use 0 (the main database).
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2191
