Commit Graph

1251 Commits

Author SHA1 Message Date
Nikita Sivukhin
05931f70ce add optional upper_bound_inclusive parameter to some checkpoint modes
- will be used in sync-engine protocol
2025-08-21 14:12:11 +04:00
PThorpe92
4a2da6c262 Remove assertion for checkpoint seq in favor of selectively using cached pages 2025-08-20 18:26:55 -04:00
PThorpe92
7082086061 Remove ENV var and enable cache by default, track which pages were cached 2025-08-20 17:42:17 -04:00
PThorpe92
345b80d14c Change env var to ENABLE instead of DISABLE so its disabled by default 2025-08-20 17:36:00 -04:00
PThorpe92
51e4cd0f1d Add debug assertion for cached pages used during checkpoint 2025-08-20 17:35:59 -04:00
PThorpe92
e28a38abc5 Fix wal tag safety issues, and add debug assertion that we are reading the proper frames 2025-08-20 17:28:48 -04:00
PThorpe92
4100737358 remove page entries without frames in frame cache in WAL rollback method 2025-08-20 17:28:19 -04:00
PThorpe92
d2c3ba14c8 Remove inefficient vec in WAL for tracking pages present in frame cache 2025-08-20 17:28:18 -04:00
PThorpe92
d6d72d2966 Update Page to carry epoch of frame + checkpont seq to ensure proper cached page for chkpt 2025-08-20 17:28:17 -04:00
PThorpe92
00f2a0f216 Performance improvements to checkpointing. prevent serializing I/O 2025-08-20 17:26:54 -04:00
PThorpe92
fe7a5e98b8 Track frame_ids on PageInner and use the page cache for reading pages to checkpoint 2025-08-20 17:24:10 -04:00
Pekka Enberg
72a5de3551 Merge 'core/mvcc: support for MVCC' from Pere Diaz Bou
This PR tries to add simple support for delete, with limited testing for
now.
Moreover, there was an error with `forward`, which wasn't obvious
without delete, which didn't skip deleted rows.

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2672
2025-08-20 16:51:31 +03:00
Jussi Saurio
e5f04ae100 Merge 'refactor/vdbe: move insert-related seeking to VDBE from BTreeCursor' from Jussi Saurio
This gets rid of `InsertState` in `BTreeCursor` plus the `moved_before`
parameter to `BTreeCursor::insert` -- instead, seek logic is now in the
existing state machines for `op_insert` and `op_idx_insert`

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2639
2025-08-20 11:15:09 +03:00
Pekka Enberg
c2208a542a Merge 'Initial pass to support per page encryption' from Avinash Sajjanshetty
This patch adds support for per page encryption. The code is of alpha
quality, was to test my hypothesis. All the encryption code is gated
behind a `encryption` flag. To play with it, you can do:
```sh
cargo run --features encryption -- database.db

turso> PRAGMA key='turso_test_encryption_key_123456';

turso> CREATE TABLE t(v);
```
Right now, most stuff is hard coded. We use AES GCM 256. This
information is not stored anywhere, but in future versions we will start
saving this info in the file. When writing to disk, we will generate a
cryptographically secure random salt, use that to encrypt the page. Then
we will store the authentication tag and the salt in the page itself. To
accommodate this encryption hardcodes reserved space of 28 bytes.
Once the key is set in the connection, we propagate that information to
pager and the WAL, to encrypt / decrypt when reading from disk.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2567
2025-08-20 11:11:24 +03:00
Avinash Sajjanshetty
40a209c000 simplify feature flag usage for encryption 2025-08-20 12:49:38 +05:30
Avinash Sajjanshetty
bd9b4bbfd2 encrypt/decrypt when writing/reading from DB 2025-08-20 11:47:23 +05:30
Avinash Sajjanshetty
657daeded3 encrypt/decrypt when writing/reading from WAL 2025-08-20 11:44:08 +05:30
Avinash Sajjanshetty
201262b3dd Update DatabaseStorage to pass encryption context 2025-08-20 11:41:08 +05:30
Avinash Sajjanshetty
94d38be1a2 Set reserved_space to 28 for encrypted databases
We will use this space to store nonce and tag
2025-08-20 11:39:09 +05:30
Avinash Sajjanshetty
a6e9237c94 Set encryption key in pager and WAL 2025-08-20 11:39:09 +05:30
Avinash Sajjanshetty
100a0d8e97 Add encryption module
Let's add an encryption module, hard coded to use AES 256 GCM.
Other required parameters are also hard coded and will be made
configurable in the future PRs.

The module is behind a `encryption` feature flag.
2025-08-20 11:38:11 +05:30
pedrocarlo
46c756b130 clear locked on pages when completion errors 2025-08-19 17:29:57 -03:00
Pere Diaz Bou
4314bc13e6 core/mvcc: delete support 2025-08-19 19:48:36 +02:00
Jussi Saurio
c2855cb0db refactor/idx_insert: move seeking to VDBE instead of BTreeCursor
Also removes `InsertState` and `moved_before` since neither are
needed anymore.
2025-08-19 19:04:42 +03:00
pedrocarlo
de1811dea7 abort completions on error 2025-08-19 10:48:21 -03:00
pedrocarlo
ab3b68e360 change completion callbacks to take a Result param + create separate functions to declare a completion errored 2025-08-19 10:48:21 -03:00
Jussi Saurio
ac37e89fe1 remove unused PAGE_ERROR flag 2025-08-19 15:44:16 +03:00
Jussi Saurio
33eb730ef8 finish_read_page() never fails, so it does not need to return Result 2025-08-19 15:44:16 +03:00
Jussi Saurio
687e593361 Add pgno field to CacheError::Locked for debugging 2025-08-19 15:44:16 +03:00
Jussi Saurio
7f1eac9560 Do not start or end transaction in nested statement 2025-08-19 13:03:14 +03:00
Jussi Saurio
9deedd917f Merge 'FaultyQuery enabled by default' from Pedro Muniz
FaultyQuery was disabled. We were not simulating faults.

Reviewed-by: Avinash Sajjanshetty (@avinassh)
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2594
2025-08-18 22:53:37 +03:00
pedrocarlo
aa7f8fbfc4 Tripwire for Wal Syncing + set syncing to true before calling sync 2025-08-18 15:34:20 -03:00
pedrocarlo
c5dc26f542 reset more pager state machines in rollback 2025-08-18 12:01:10 -03:00
Pere Diaz Bou
94cd504d4c core/mvcc: support schema_did change on commit_txn
This not only changes schema_did_change on commit_txn for mvcc, but also
extracts the connection transaction state from non mvcc transactions to
mvcc too.
2025-08-18 15:52:10 +02:00
Jussi Saurio
9abc63d853 Add a bit of abstraction for creating EQ cursorcontexts 2025-08-18 13:13:02 +03:00
Jussi Saurio
3eb89982ba Remove obsolete FIXME 2025-08-18 12:08:40 +03:00
Jussi Saurio
50fd7ec58b Refactor: use regular save/restore context mechanism for delete balancing
- Removes special `DeleteSavepoint` and uses the existing cursor restoration
  mechanism.
- This required some restructuring of `DeleteState` to avoid cloning it, i.e.
  some negotations with the borrow checker.
- CursorContext now takes a SeekOp as well to allow retaining the behavior
  that we use LT for seeking after a delete-induced rebalancing. This behavior
  will probably be removed as part of fixing #2004, but here I am trying to
  preserve the current semantics.
2025-08-18 11:58:00 +03:00
PThorpe92
605b3e233f Move in memory reset logic to before truncation in wal restart 2025-08-16 17:42:22 -04:00
PThorpe92
eae68f3669 Update stale in memory wal header after restarting log 2025-08-16 17:36:48 -04:00
PThorpe92
2c526c4c37 Add io_yield_x macros to reduce boilerplate 2025-08-16 16:14:00 -04:00
Nikita Sivukhin
71bbc36f61 adjust log message 2025-08-15 13:19:46 +04:00
Nikita Sivukhin
2361587d81 cargo fmt 2025-08-15 13:18:50 +04:00
Nikita Sivukhin
9b08f8c0f2 set in-memory WAL information according to last commited frame
- before that pages_in_frames, frame_cache and last_checksum was set to the latest written (not commited!) frame in the WAL found on disk
2025-08-15 13:13:33 +04:00
Pekka Enberg
e4f424f9e6 Merge 'Properly implement CLI command' from Preston Thorpe
Closes #2588
SQLite internally implements `.clone` by doing something like piping
`.dump` into a new connection to a database attached to the file you
want. This PR implements that by adding an `ApplyWriter` that implements
`std::fmt::Write`, and refactors our current `.dump` plumbing to work
with any `Write` interface, so we can `.dump` to stdout or `.dump` to a
new connection, therefore cloning the database.

Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2590
2025-08-15 10:44:57 +03:00
PThorpe92
cc2fed3297 Remove copy_to API from file IO trait 2025-08-14 21:31:13 -04:00
PThorpe92
3c088dda59 Update callsites of copy_to Database impl 2025-08-14 21:31:13 -04:00
PThorpe92
2ad479f413 dont forget other instance where db file is expected size 2025-08-14 19:56:23 -04:00
PThorpe92
9017acd622 Ensure we fsync the db file in all paths that we checkpoint 2025-08-14 19:23:00 -04:00
PThorpe92
5a7c9325e2 Add assertion for expected write amount in writev callback 2025-08-14 10:53:16 -04:00
Jussi Saurio
bd8c6f3c7c make PageSize more robust: only accept literal '1' value if it comes directly from db header 2025-08-14 12:40:58 +03:00