`sqlite3.h`
```c
/* Usually, when a database in [WAL mode] is closed or detached from a
** database handle, SQLite checks if there are other connections to the
** same database, and if there are no other database connection (if the
** connection being closed is the last open connection to the database),
** then SQLite performs a [checkpoint] (in truncate mode) before closing the connection and
** deletes the WAL file.
...
```
Currently, the WAL grows unbounded. and because we don't have a `shm`
file, we do not trust `nbackfills`, and we read (and backfill) the
entire WAL every time we open it. So unless there is a manual `PRAGMA
wal_checkpoint(truncate);` issued by a user, this will severely degrade
performance, at least for the first cacheflush each time a database is
opened.
SQLite, when closing the final connection, will automatically run a
checkpoint in truncate mode. We should do this as well :)
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2761
Things that were just wrong:
1. No pages other than the root page were checked, because no looping
was done. Add a loop.
2. Rightmost child page was never added to page stack. Add it.
New integrity check features:
- Add overflow pages to stack as well
- Check that no page is referenced more than once in the tree
Closes#2781
This PR adds a method `append_frames_vectored` that takes N frames and
optionally the `db size` which will need to be set for the last (commit)
frame, and it calculates the checksums and submits them as a single
`pwritev` call, drastically reducing the number of syscalls needed for
each write operation.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2751
My goal with this patch is to be able to implement the ProjectOperator
for DBSP circuits using VDBE for expression evaluation.
*not* doing so is dangerous for the following reason: we will end up
with different, subtle, and incompatible behavior between SQLite
expressions if they are used in views versus outside of views.
In fact, even in our prototype had them: our projection tests, which
used to pass, were actually wrong =) (sqlite would return something
different if those functions were executed outside the view context)
For optimization reasons, we single out trivial expressions: they don't
have go through VDBE. Trivial expressions are expressions that only
involve Columns, Literals, and simple operators on elements of the same
type. Even type coercion takes this out of the realm of trivial.
Everything that is not trivial, is then translated with translate_expr -
in the same way SQLite will, and then compiled with VDBE.
We can, over time, make this process much better. There are essentially
infinite opportunities for optimization here. But for now, the main
warts are:
* VDBE execution needs a connection
* There is no good way in VDBE to pass parameters to a program.
* It is almost trivial to pollute the original connection. For example,
we need to issue HALT for the program to stop, but seeing that halt
will usually cause the program to try and halt the original program.
Subprograms, like the ones we use in triggers are a possible solution,
but they are much more expensive to execute, especially given that our
execution would essentially have to have a program with no other role
than to wrap the subprogram.
Therefore, what I am doing is:
* There is an in-memory database inside the projection operator (an
obvious optimization is to share it with *all* projection operators).
* We obtain a connection to that database when the operator is created
* We use that connection to execute our VDBE, which offers a clean, safe
and isolated way to execute the expression.
* We feed the values to the program manually by editing the registers
directly.
To be used in DBSP-based projections. This will compile an expression
to VDBE bytecode and execute it.
To do that we need to add a new type of Expression, which we call a
Register.
This is a way for us to pass parameters to a DBSP program which will be
not columns or literals, but inputs from the DBSP deltas.
We have an issue at the moment that when a materialized view fails
to be created, we just swallow the error and leave the database in
a funny state.
We have can_create_view() to detect those issues early, but not all
errors can be detected that early.
Things that were just wrong:
1. No pages other than the root page were checked, because no looping
was done. Add a loop.
2. Rightmost child page was never added to page stack. Add it.
New integrity check features:
- Add overflow pages to stack as well
- Check that no page is referenced more than once in the tree
In addition to the existing `append_frame` which will write an individual frame
to the WAL, we add a method `append_frames_vectored` that takes N frames and the
db size which will need to be set for the last (commit) frame, and it
calculates the checksums and submits them as a single `pwritev` call,
reducing the number of syscalls needed for each write operation.
Commit ebe6aa0d28 ("adjust cfg for unix
and linux IO") adjusted the I/O conditional compilation, but forgot that
Android and iOS are also part of Unix target family.
Fixes#2500Closes#2776
No need to pass `disable` flag to the `end_tx` method as it has that
info from connection itself
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2777
- Transaction which was started with max_frame = 0 and
max_frame_read_lock_index = 0 can write to the WAL and in this case it
needs to read data back from WAL and not the DB file.
- Without cache spilling its hard to reproduce this issue for the turso-
db now, but I found this issue with sync-engine which do weird stuff
with the WAL which "simulates" cache spilling behaviour to some extent.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2735
Commit ebe6aa0d28 ("adjust cfg for unix
and linux IO") adjusted the I/O conditional compilation, but forgot that
Android and iOS are also part of Unix target family.
Fixes#2500
- transaction which was started with max_frame = 0 and max_frame_read_lock_index = 0
can write to the WAL and in this case it needs to read data back from WAL
- without cache spilling its hard to reproduce this issue for the turso-db now,
but I stumbled into this issue with sync-engine which do weird stuff with the WAL
which "simulates" cache spilling behaviour to some extent
Closes#2738Closes#2739Closes#2753Closes#2755Closes#2767Closes#2768Closes#2769Closes#2770
## El problema
If a connection does e.g. CREATE TABLE, it will start a "child
statement" to reparse the schema. That statement does not start its own
transaction, and so should not try to end the existing one either.
We had a logic bug where these steps would happen:
- `CREATE TABLE` executed successfully
- pread fault happens inside `ParseSchema` child stmt
- `handle_program_error()` is called
- `pager.end_tx()` returns immediately because `is_nested_stmt` is true
and we correctly no-op it.
- however, crucially: `handle_program_error()` then sets tx state to
None
- parent statement now catches error from nested stmt and calls
`handle_program_error()`, which calls `pager.end_tx()` again, and since
txn state is None, when it calls `rollback()` we panic on the assertion
`"dirty pages should be empty for read txn"`
## La solucion
Do not do _any_ error processing in `handle_program_error()` inside a
nested stmt. This means that the parent write txn is still active when
it processes the error from the child and we avoid this panic.
Closes#2772
i had a rough time reading this function earlier and trying to
understand it, so rewrote it in a way that, to me, is much more
readable.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2509
`make test` fails when using the nix-shell environment due to `uv` not
being included in the list of dependencies with
`make: uv: No such file or directory
make: *** [Makefile:55: uv-sync-test] Error 127`
simple fix adding it to `nativeBuildInputs` of the shell. after that
runs as expected.
Reviewed-by: Levy A. (@levydsa)
Closes#2636
This PR removes `Result<()>` from `Jsonb::write_to_string()`, since it
wasn't required. The method now returns `()`.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2717
If a connection does e.g. CREATE TABLE, it will start a "child statement"
to reparse the schema. That statement does not start its own transaction,
and so should not try to end the existing one either.
We had a logic bug where these steps would happen:
- `CREATE TABLE` executed successfully
- pread fault happens inside `ParseSchema` child stmt
- `handle_program_error()` is called
- `pager.end_tx()` returns immediately because `is_nested_stmt` is true
and we correctly no-op it.
- however, crucially: `handle_program_error()` then sets tx state to None
- parent statement now catches error from nested stmt and calls
`handle_program_error()`, which calls `pager.end_tx()` again, and since
txn state is None, when it calls `rollback()` we panic on the assertion
`"dirty pages should be empty for read txn"`
Solution:
Do not do _any_ error processing in `handle_program_error()` inside a nested
stmt. This means that the parent write txn is still active when it processes
the error from the child and we avoid this panic.
