Previously, the encryption module had hardcoded a lot of things. This
refactor makes it slightly nice and makes it configurable.
Right now cipher algorithm is assumed and hardcoded, I will make that
configurable in the upcoming PR
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2722
Problems:
1. fill_cell_payload() is not re-entrant because it can yield IO
on allocating a new overflow page, resulting in losing some of the
input data.
2. fill_cell_payload() in its current form is not safe for cache spilling
because the previous overflow page in the chain of allocated overflow pages
can be evicted by a spill caused by the next overflow page allocation,
invalidating the page pointer and causing corruption.
3. fill_cell_payload() uses raw pointers and `unsafe` as a workaround from a previous time when we used to clone `WriteState`, resulting in hard-to-read code.
Solutions:
1. Introduce a new substate to the fill_cell_payload state machine to handle
re-entrancy wrt. allocating overflow pages.
2. Always pin the current overflow page so that it cannot be evicted during the
overflow chain construction. Also pin the regular page the overflow chain is
attached to, because it is immediately accessed after fill_cell_payload is done.
3. Remove all explicit usages of `unsafe` from `fill_cell_payload` (although our pager is ofc still extremely unsafe under the hood :] )
Note that solution 2 addresses a problem that arose in the development of page cache
spilling, which is not yet implemented, but will be soon.
Miscellania:
1. Renamed a bunch of variables to be clearer
2. Added more comments about what is happening in fill_cell_payload
I'm working on ANALYZE. I'm using EXPLAIN. The lack of highlighting
for them in the CLI annoyed me a bit.
I don't think there's any tests for this? I'm mostly at a "it seems to
work for me". I double checked that `EXPLAIN SELECT CASE 0 WHEN 0 THEN
0 ELSE 1` syntax highlights, to make sure I didn't break the longer
parsing (which I had).
Closes#2741
This PR make it possible to do 2 pretty crazy things with turso-db:
1. Now we can mix WAL frames inserts with SQL execution within same
transaction. This will allow sync engine to execute rebase of local
changes within atomically over main database file (the operation first
require us to push new frames to physically revert local changes and
then we need to replay local logical changes on top of the modified DB
state)
2. Under `conn_raw_api` Cargo feature turso-db now expose method which
allow caller to specify WAL file path. This dangerous capability exposed
for sync-engine which maintain 2 databases: main one and "revert"-DB
which shares same DB file but has it's own separate WAL. As sync-engine
has full control over checkpoint - it can guarantee that DB file will be
consistent with both main and "revert" DB WALs.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2716
Closes#2715
1. Since our multithreading isn't proven correct (and is actually
probably all kinds of incorrect), let's serialize these tests since they
operate on the same database
2. Use `lock()` instead of `try_lock()` - i.e. wait to obtain the lock
on the file instead of immediately erroring if we can't
Closes#2729
This kind of fault does not semantically represent anything real, since
we already have fault injection for every concrete IO operation like
reading, writing, syncing and so forth.
Moreover, having this feature is the direct cause of the false positive
simulator failure as reported in issue #2727. There, a "run_once fault"
happened immediately after we fsynced following an INSERT, which caused
the simulator to think the INSERT failed, and later a sim assertion
failed because the on-disk database had 1 more row than it thought it
would.
Closes#2727
Reviewed-by: Pekka Enberg <penberg@iki.fi>
Closes#2728
This kind of fault does not semantically represent anything real,
since we already have fault injection for every concrete IO operation
like reading, writing, syncing and so forth.
Moreover, having this feature is the direct cause of the false positive
simulator failure as reported in issue #2727. There, a "run_once fault"
happened immediately after we fsynced following an INSERT, which caused
the simulator to think the INSERT failed, and later a sim assertion failed
because the on-disk database had 1 more row than it thought it would.
Add `truncate` method in the page cache which remove all entries which
reference pages greater than new DB size.
This will be used in the sync engine as in its case DB size can shrink
when we "rebase" changes from remote to local.
It stands on the #2707 because touch few files from that PR
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2711
