## What does this fix
This PR fixes an issue with BTree upwards traversal logic where we would
try to go up to a parent node in `next()` even though we are at the very
end of the btree. This behavior can leave the cursor incorrectly
positioned at an interior node when it should be at the right edge of
the rightmost leaf.
## Why doesn't it cause problems on main
This bug is masked on `main` by every table `insert()` (wastefully)
calling `find_cell()`:
- `op_new_rowid` called, let's say the current max rowid is `666`.
Cursor is left pointing at `666`.
- `insert()` is called with rowid `667`, cursor is currently pointing at
`666`, which is incorrect.
- `find_cell()` does a binary search every time, and hence somewhat
accidentally positions the cursor correctly _after_ `666` so that the
insert goes to the correct place
## Why was this issue found
in #1988, I am removing `find_cell()` entirely in favor of always
performing a seek to the correct location - and skipping `seek` when it
is not required, saving us from wasting a binary search on every insert
- but this change means that we need to call `next()` after
`op_new_rowid` to have the cursor positioned correctly at the new
insertion slot. Doing this surfaces this upwards traversal bug in that
PR branch.
## Details of solution
- Store `cell_count` together with `cell_idx` in pagestack, so that
chlidren can know whether their parents have reached their end without
doing IO
- To make this foolproof, pin pages on `PageStack` so the page cache
cannot evict them during tree traversal
- `cell_indices` renamed to `node_states` since it now carries more
information (cell index AND count, instead of just index)
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2005
This PR addresses https://github.com/tursodatabase/turso/issues/1828 in
a phased manner.
Making database header access async in one PR will be complicated. This
PR ports adds an async API to `header_accessor.rs` and ports over some
of `pager.rs` to use this API.
This will allow gradual porting over of all call sites. Once all call
sites are ported over, one mechanical rename will fix everything in the
repo so we don't have any `<header_name>_async` functions.
Also, porting header accessors over from sync to async would be a good
way to get introduced to the Limbo codebase for first time contributors.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1966
`db_size` is `>0` in case of last frame written of a transaction. This
is necessary as we need to know -- while recovering wal contents -- that
we have read a transaction fully instead of treating every frame as its
own transaction.
### Problem
Profiling revealed that `usable_space()` calls were consuming 60% of
total execution time for simple SELECT queries, making Limbo
approximately `6x` slower than SQLite for SELECT operations.
The bottleneck was caused by `usable_space()` performing expensive I/O
operations on every call to read `page_size` and `reserved_space` from
the database header, despite `page_size` values being effectively
immutable after database initialization. Only `reserved_space` is
allowed to increase in SQLite.
Evidence: https://share.firefox.dev/44tCUIy
### Solution
Implemented OnceCell-based caching for both page_size and reserved_space
values in the Pager struct:
`page_size: OnceCell<u16>` - Page size is immutable after database
initialization per SQLite specification
`reserved_space: OnceCell<u8>` - Reserved space rarely changes and only
grows, safe to cache
### Performance Impact
Benchmark results: Simple SELECT query time reduced from ~2.89ms to
~1.29ms (~55% improvement)
Closes#1852
Support for simple interactive rollback like:
```sql
create table t (x);
insert into t values (1);
begin;
insert into t values (2);
rollback;
select * from t;
```
This PR also fixes some other issues I found while debugging:
* Checkpoint would never `clear_dirty` on pages in page cache.
* Auto commit for interactive transactions was not respected so any
`insert` after `begin` would flush frames regardless of `auto_commit`
state.
* `max_frame` on wal shared state was being updated after every
`append_frame` which was incorrect, as another transaction would be able
to use that new `max_frame` even tho the transaction could've rolled
back. Instead we update the private copy of `max_frame` and only update
it at the end.
Follow up for later are savepoints which require implementing a
subjournal to track savepoints and their modified pages.
Closes#1825
When `struct Database` is constructed, store `is_empty` as an
`Arc<AtomicBool>` - the value is true if:
1. DB size is zero
2. WAL has no frames
When `struct Pager` is constructed, this `Arc` is simply cloned.
When any connection runs a transaction it will first check `is_empty`,
and if the DB is empty, it will lock `init_lock` and then check `is_empty`
again, and if it's still true, it allocates page1 and stores `false` in
the `is_empty` `AtomicBool` and drops the lock.
---
Note that Limbo can currently have a zero DB and a WAL with frames,
as we have no special logic for folding page1 to the main DB file
during initialization.
Page 1 allocation currently happens on the first transaction (read or
write, due to having to support `select * from sqlite_schema` on an
empty DB; we should really check how SQLite actually does this.).
