When we reverted the #2789 , we had already merged #2793 . That Pr used
some helper methods that were created in #2789. So I just added them
back here + fixed the simulator dockerfile.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2803
We had missed #2794 because we don't get any visible feedback on PRs if
we return wrong results from TPC-H, so let's make that happen here.
Closes#2798
Decouple Sql generation code from simulator code, so that it can
potentially be reused for fuzzing on other crates and to create a
`GenerationContext` trait so that it becomes easier to create
`Simulation Profiles`. Ideally in further PRs, I want to expand the
`GenerationContext` trait so we can guide the generation with context
from the simulation profile.
Depends on #2789 .
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2793
Fixes TPC-H query 13 from returning an incorrect result. In this
specific case, we were returning non-null `IdxRowid` values for the
right-hand side table even when there was no match with the left-hand
side table, meaning the join produced matches even in cases where there
shouldn't have been any.
Closes#2794Closes#2795
Fixes TPC-H query 13 from returning an incorrect result. In this specific
case, we were returning non-null `IdxRowid` values for the right-hand side
table even when there was no match with the left-hand side table, meaning
the join produced matches even in cases where there shouldn't have been any.
Closes#2794
## Make fill_cell_payload() safe for async IO and cache spilling
### Problems:
1. fill_cell_payload() is not re-entrant because it can yield IO
on allocating a new overflow page, resulting in losing some of the
input data.
2. fill_cell_payload() in its current form is not safe for cache
spilling
because the previous overflow page in the chain of allocated overflow
pages
can be evicted by a spill caused by the next overflow page
allocation,
invalidating the page pointer and causing corruption.
3. fill_cell_payload() uses raw pointers and `unsafe` as a workaround
from a previous time when we used to clone `WriteState`, resulting in
hard-to-read code.
### Solutions:
1. Introduce a new substate to the fill_cell_payload state machine to
handle
re-entrancy wrt. allocating overflow pages.
2. Always pin the current overflow page so that it cannot be evicted
during the
overflow chain construction. Also pin the regular page the overflow
chain is
attached to, because it is immediately accessed after
fill_cell_payload is done.
3. Remove all explicit usages of `unsafe` from `fill_cell_payload`
(although our pager is ofc still extremely unsafe under the hood :] )
Note that solution 2 addresses a problem that arose in the development
of page cache
spilling, which is not yet implemented, but will be soon.
### Miscellania:
1. Renamed a bunch of variables to be clearer
2. Added more comments about what is happening in fill_cell_payload
Closes#2737
Generic IO and Windows IO were identical, since we don't do anything
windows specific (maybe when someone eventually wants to implement an IO
back-end for windows `IO_RING` API, we can bring it back :)
This does use the exact impl of `WindowsIO`, simply renaming it to
Generic IO.. because I don't think GenericIO had ever been used.
Reviewed-by: Pedro Muniz (@pedrocarlo)
Closes#2790
This patch brings a bunch of quality of life improvements to encryption:
1. Previously, we just let any string to be used as a key. I have
updated the `PRAGMA hexkey=''` to get the key in hex. I have also
renamed from `key`, because that will be used to get passphrase
2. Added `PRAGMA cipher` so that now users can select which cipher they
want to use (for now, either `aegis256` or `aes256gcm`)
3. We now set the encryption context when both cipher and key are set
I also updated tests to reflect this.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2779
Sometimes in the parser we had enum Variants that contained
`Vec<Box<Expr>>` which is unnecessary and inefficient.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2789
Resolves#2778.
I think this issue is probably because backticks in shell end up causing
the command-line to attempt to resolve whatever was inside it as a
command.
Added `shlex.quote` to the commit title in order to hopefully fix this.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2788
`sqlite3.h`
```c
/* Usually, when a database in [WAL mode] is closed or detached from a
** database handle, SQLite checks if there are other connections to the
** same database, and if there are no other database connection (if the
** connection being closed is the last open connection to the database),
** then SQLite performs a [checkpoint] (in truncate mode) before closing the connection and
** deletes the WAL file.
...
```
Currently, the WAL grows unbounded. and because we don't have a `shm`
file, we do not trust `nbackfills`, and we read (and backfill) the
entire WAL every time we open it. So unless there is a manual `PRAGMA
wal_checkpoint(truncate);` issued by a user, this will severely degrade
performance, at least for the first cacheflush each time a database is
opened.
SQLite, when closing the final connection, will automatically run a
checkpoint in truncate mode. We should do this as well :)
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2761
Things that were just wrong:
1. No pages other than the root page were checked, because no looping
was done. Add a loop.
2. Rightmost child page was never added to page stack. Add it.
New integrity check features:
- Add overflow pages to stack as well
- Check that no page is referenced more than once in the tree
Closes#2781
This PR adds a method `append_frames_vectored` that takes N frames and
optionally the `db size` which will need to be set for the last (commit)
frame, and it calculates the checksums and submits them as a single
`pwritev` call, drastically reducing the number of syscalls needed for
each write operation.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2751
My goal with this patch is to be able to implement the ProjectOperator
for DBSP circuits using VDBE for expression evaluation.
*not* doing so is dangerous for the following reason: we will end up
with different, subtle, and incompatible behavior between SQLite
expressions if they are used in views versus outside of views.
In fact, even in our prototype had them: our projection tests, which
used to pass, were actually wrong =) (sqlite would return something
different if those functions were executed outside the view context)
For optimization reasons, we single out trivial expressions: they don't
have go through VDBE. Trivial expressions are expressions that only
involve Columns, Literals, and simple operators on elements of the same
type. Even type coercion takes this out of the realm of trivial.
Everything that is not trivial, is then translated with translate_expr -
in the same way SQLite will, and then compiled with VDBE.
We can, over time, make this process much better. There are essentially
infinite opportunities for optimization here. But for now, the main
warts are:
* VDBE execution needs a connection
* There is no good way in VDBE to pass parameters to a program.
* It is almost trivial to pollute the original connection. For example,
we need to issue HALT for the program to stop, but seeing that halt
will usually cause the program to try and halt the original program.
Subprograms, like the ones we use in triggers are a possible solution,
but they are much more expensive to execute, especially given that our
execution would essentially have to have a program with no other role
than to wrap the subprogram.
Therefore, what I am doing is:
* There is an in-memory database inside the projection operator (an
obvious optimization is to share it with *all* projection operators).
* We obtain a connection to that database when the operator is created
* We use that connection to execute our VDBE, which offers a clean, safe
and isolated way to execute the expression.
* We feed the values to the program manually by editing the registers
directly.
To be used in DBSP-based projections. This will compile an expression
to VDBE bytecode and execute it.
To do that we need to add a new type of Expression, which we call a
Register.
This is a way for us to pass parameters to a DBSP program which will be
not columns or literals, but inputs from the DBSP deltas.
We have an issue at the moment that when a materialized view fails
to be created, we just swallow the error and leave the database in
a funny state.
We have can_create_view() to detect those issues early, but not all
errors can be detected that early.
Things that were just wrong:
1. No pages other than the root page were checked, because no looping
was done. Add a loop.
2. Rightmost child page was never added to page stack. Add it.
New integrity check features:
- Add overflow pages to stack as well
- Check that no page is referenced more than once in the tree
