Revives the `MemorySim` PR and fixes a page cache issue where we could
have a unlocked and unloaded page in the page cache after a FaultyQuery.
The page would continue in the cache and could affect other queries as
the `page_cache` is at the `Connection` level.
Depends on #2785Closes#2693
## Changes
- Refactor sql generation to always accept a `Context` trait object so
we can query the current Generation `Opts`. This change allows us to be
more granular in generating our sql statements. It also opens
opportunities for us to add even more knobs to tweak generation as
needed. I tried to make this as generic as possible as I believe this
library can be useful for fuzz testing outside the simulator.
- Introduce `Profile` struct that aggregates the different
configurations needed to execute the simulator. With this Profile struct
we can bias sql generation towards different statements and create
predefined profiles.
`WriteHeavy` Profile:
```rust
Profile {
query: QueryProfile {
gen_opts: Opts {
// TODO: in the future tweak blob size for bigger inserts
// TODO: increase number of rows as well
table: TableOpts {
large_table: LargeTableOpts {
large_table_prob: 0.4,
..Default::default()
},
..Default::default()
},
query: QueryOpts {
insert: InsertOpts {
min_rows: NonZeroU32::new(5).unwrap(),
max_rows: NonZeroU32::new(11).unwrap(),
},
..Default::default()
},
..Default::default()
},
select_weight: 30,
insert_weight: 70,
delete_weight: 0,
update_weight: 0,
..Default::default()
},
..Default::default()
};
```
As you can see we disable the `delete` and `update` weights, decrease
`select` and increase `insert` weights. This means that we disable
updates and deletes in favor of inserting more data and checking the
validity of the database with fewer select statements.
- `Profile` and `Opts` are validated with `garde` and can generate json
schemas with `schemars` so that we can have editor integration when
creating new profiles to play with.
- Added some docs in the README explaining how you can add LSP
integration for the Json config by generating a `JsonSchema` file
Closes#2852
This patch adds support for per page encryption. The code is of alpha
quality, was to test my hypothesis. All the encryption code is gated
behind a `encryption` flag. To play with it, you can do:
```sh
cargo run --features encryption -- database.db
turso> PRAGMA key='turso_test_encryption_key_123456';
turso> CREATE TABLE t(v);
```
Right now, most stuff is hard coded. We use AES GCM 256. This
information is not stored anywhere, but in future versions we will start
saving this info in the file. When writing to disk, we will generate a
cryptographically secure random salt, use that to encrypt the page. Then
we will store the authentication tag and the salt in the page itself. To
accommodate this encryption hardcodes reserved space of 28 bytes.
Once the key is set in the connection, we propagate that information to
pager and the WAL, to encrypt / decrypt when reading from disk.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2567
Let's add an encryption module, hard coded to use AES 256 GCM.
Other required parameters are also hard coded and will be made
configurable in the future PRs.
The module is behind a `encryption` feature flag.
This PR switches from usage of WAL push to SQL over http calls.
As we have sequence of logical changes from CDC - we can just send SQL
statements to the remote. This will give us benefits of more smooth
concurrent usage of sync - because WAL push is very unfriendly in case
of concurrent usages.
Closes#2617
- otherwise, in case of schema change, connection will constantly get Database schema is changed error as reprepare logic is implemented in the statement event loop
Use the same rusqlite version in every crate and use a bundled up-to-
date sqlite version
(the impetus for this PR is still me trying to figure out why sqlite in
the insert benchmark doesn't seem to be fsyncing, even when instructed)
Closes#2507
We should be allocator-agnostic. It is pretty limiting for us to force a
user to use a particular allocator. This is specially restricting for
`no_std` in the future.
Reviewed-by: bit-aloo (@Shourya742)
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2481
