Add `truncate` method in the page cache which remove all entries which
reference pages greater than new DB size.
This will be used in the sync engine as in its case DB size can shrink
when we "rebase" changes from remote to local.
It stands on the #2707 because touch few files from that PR
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2711
The Unix backend is a syscall()-based, blocking implementation. The
O_NONBLOCK adds nothing.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2708
This PR adds information about checkpoint sequence number to the WAL raw
API. Will be used in the sync engine.
Depends on the #2699
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2707
SQLite does not store the rowid alias column in the record at all
when it is a rowid alias, because the rowid is always stored anyway
in the record header.
Found when running simulator in #2641
All indexes store the rowid as the last column, so whenever the rowid of
a given row changes the index entry must also be deleted and reinserted
with the new index.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2712
UPDATE should skip over the UNIQUE constraint failure if the existing
row it found during the check has the same rowid as the row we are
currently updating
Same deal as #2700, except this time in UPDATE. Nothing tests this on
`main` so not caught.
I will later put #2641 into mergeable condition so it will catch all of
these going forward.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2710
Previously, we just hardcoded the reserved space with encryption flag.
This patch removes that and sets the reserved space if a key was
specified during a creation of db
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2706
Previously, we just hardcoded the reserved space with encryption flag.
This patch removes that and sets the reserved space if a key was
specified during a creation of db
### General idea:
(outside of other optimizations made mostly around concurrency):
**When checkpointing, use pages from the PageCache if we can determine
that they are exactly the page/frame that we want.**
e.g. if the frame_cache has an entry:
`Page ID: 104 -> Frame ID's: [1001, 1002]`
and the OngoingCheckpoint has min_frame of 999 and max_frame of 1020, we
should be able to check the PageCache and see if it has page 104, and
only if it is tagged with frame_id = 1002, can we use that page to
backfill the DB file.
Since using a cached page during checkpoint is purely an optimization,
we can be conservative in terms of when we accept that a cached page is
valid to use. I came up with a `wal_tag` which is the frame_id +
checkpoint_seq, which is set only in the two following places:
1. When explicitly reading a frame from the WAL. (inside
Wall::read_frame)
- read_frame is perhaps the most obvious path of ensuring it's the
exact page + frame combination that we want.
2. When appending a frame to the log during the normal process of
writing (during `[Pager::cacheflush]`)
- cacheflush calls append_frame, and inside the Completion, the dirty
flag is cleared, and the wal_tag flag is set to the frame_id.
Inside `finish_read_page` (which is called for every page we read from
either the DB file or WAL.. the `wal_tag` is cleared along with the
`dirty` flag, so that any re-used `PageRef's` don't contain wal_tag's
from any previous or stale pages.
#### **Proposal**:
(In order to merge and simultaneously be able to sleep at night)
there is this debug assertion:
```rust
#[cfg(debug_assertions)]
{
let mut raw = vec![0u8; self.page_size() as usize + WAL_FRAME_HEADER_SIZE];
self.io.wait_for_completion(self.read_frame_raw(target_frame, &mut raw)?)?;
let (_, wal_page) = sqlite3_ondisk::parse_wal_frame_header(&raw);
let cached = cached_page.get_contents().buffer.as_slice();
// while being horrible for performance, we can ensure that the bytes are identical
// when using the cached page vs what we would otherwise have read from disk.
turso_assert!(wal_page == cached, "cache fast-path returned wrong content for page {page_id} frame {target_frame}");
}
```
Performance
=====================================
Average latency for a checkpoint on my local machine:
#### Before: `7-12ms`
#### After: `2-5ms`
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2568
