SQLite does not store the rowid alias column in the record at all
when it is a rowid alias, because the rowid is always stored anyway
in the record header.
Found when running simulator in #2641
All indexes store the rowid as the last column, so whenever the rowid of
a given row changes the index entry must also be deleted and reinserted
with the new index.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2712
UPDATE should skip over the UNIQUE constraint failure if the existing
row it found during the check has the same rowid as the row we are
currently updating
Same deal as #2700, except this time in UPDATE. Nothing tests this on
`main` so not caught.
I will later put #2641 into mergeable condition so it will catch all of
these going forward.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2710
Previously, we just hardcoded the reserved space with encryption flag.
This patch removes that and sets the reserved space if a key was
specified during a creation of db
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2706
This PR introduces optional upper_bound for PASSIVE and TRUNCATE
checkpoint modes
This is needed for sync engine where we need to have control over WAL:
1. TRUNCATE with upper_bound used as a way to checkpoint WAL only if
there were no frames written since sync-engine read max_frame_no
2. PASSIVE with upper_bound used to checkpoint only certain prefix of
the WAL
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2699
Previously, we just hardcoded the reserved space with encryption flag.
This patch removes that and sets the reserved space if a key was
specified during a creation of db
### General idea:
(outside of other optimizations made mostly around concurrency):
**When checkpointing, use pages from the PageCache if we can determine
that they are exactly the page/frame that we want.**
e.g. if the frame_cache has an entry:
`Page ID: 104 -> Frame ID's: [1001, 1002]`
and the OngoingCheckpoint has min_frame of 999 and max_frame of 1020, we
should be able to check the PageCache and see if it has page 104, and
only if it is tagged with frame_id = 1002, can we use that page to
backfill the DB file.
Since using a cached page during checkpoint is purely an optimization,
we can be conservative in terms of when we accept that a cached page is
valid to use. I came up with a `wal_tag` which is the frame_id +
checkpoint_seq, which is set only in the two following places:
1. When explicitly reading a frame from the WAL. (inside
Wall::read_frame)
- read_frame is perhaps the most obvious path of ensuring it's the
exact page + frame combination that we want.
2. When appending a frame to the log during the normal process of
writing (during `[Pager::cacheflush]`)
- cacheflush calls append_frame, and inside the Completion, the dirty
flag is cleared, and the wal_tag flag is set to the frame_id.
Inside `finish_read_page` (which is called for every page we read from
either the DB file or WAL.. the `wal_tag` is cleared along with the
`dirty` flag, so that any re-used `PageRef's` don't contain wal_tag's
from any previous or stale pages.
#### **Proposal**:
(In order to merge and simultaneously be able to sleep at night)
there is this debug assertion:
```rust
#[cfg(debug_assertions)]
{
let mut raw = vec![0u8; self.page_size() as usize + WAL_FRAME_HEADER_SIZE];
self.io.wait_for_completion(self.read_frame_raw(target_frame, &mut raw)?)?;
let (_, wal_page) = sqlite3_ondisk::parse_wal_frame_header(&raw);
let cached = cached_page.get_contents().buffer.as_slice();
// while being horrible for performance, we can ensure that the bytes are identical
// when using the cached page vs what we would otherwise have read from disk.
turso_assert!(wal_page == cached, "cache fast-path returned wrong content for page {page_id} frame {target_frame}");
}
```
Performance
=====================================
Average latency for a checkpoint on my local machine:
#### Before: `7-12ms`
#### After: `2-5ms`
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2568
We were storing `txid` in `ProgramState`, this meant it was impossible
to track interactive transactions. This was extracted to `Connection`
instead.
Moreover, transaction state for mvcc now is reset on commit.
Closes#2689
This PR tries to add simple support for delete, with limited testing for
now.
Moreover, there was an error with `forward`, which wasn't obvious
without delete, which didn't skip deleted rows.
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#2672
