Commit Graph

341 Commits

Author SHA1 Message Date
Pekka Enberg
13e62ce435 Merge 'core: Initial pass on synchronous pragma' from Pekka Enberg
This adds support for "OFF" and "FULL" (default) synchronous modes. As
future work, we need to add NORMAL and EXTRA as well because
applications expect them.

Closes #2833
2025-08-29 07:27:12 +03:00
Jussi Saurio
9aae3fa859 refactor: remove BTreePageInner
it wasn't used for anything. no more `page.get().get().id`.
2025-08-28 21:44:54 +03:00
Pekka Enberg
44ed4d562f core: Initial pass on synchronous pragma
This adds support for "OFF" and "FULL" (default) synchronous modes. As
future work, we need to add NORMAL and EXTRA as well because
applications expect them.
2025-08-28 16:02:41 +03:00
Avinash Sajjanshetty
2c0842ff52 Set and propagate IOContext as required 2025-08-27 22:05:01 +05:30
Pekka Enberg
2921033b28 core/storage: Micro-optimize Pager::commit_dirty_pages()
There's no need to call io.now() unless debug tracing is on. Let's
micro-optimize commit_dirty_pages() to avoid the unnecessary call.
2025-08-27 11:12:43 +03:00
Jussi Saurio
bf58d179db Improve documentation of page pinning 2025-08-26 10:13:25 +03:00
Pekka Enberg
8f11311473 Merge 'Improve encryption API' from Avinash Sajjanshetty
This patch brings a bunch of quality of life improvements to encryption:
1. Previously, we just let any string to be used as a key. I have
updated the `PRAGMA hexkey=''` to get the key in hex. I have also
renamed from `key`, because that will be used to get passphrase
2. Added `PRAGMA cipher` so that now users can select which cipher they
want to use (for now, either `aegis256` or `aes256gcm`)
3. We now set the encryption context when both cipher and key are set
I also updated tests to reflect this.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2779
2025-08-26 08:32:29 +03:00
PThorpe92
2d661e3304 Apply review suggestions, add logging 2025-08-25 16:56:43 -04:00
PThorpe92
1b514e6d0f Only checkpoint final remaining DB connection, and use Truncate mode 2025-08-25 16:52:29 -04:00
PThorpe92
37a7ec7477 Update append_frames_vectored to use new encryption_ctx and apply review 2025-08-25 09:50:57 -04:00
PThorpe92
daea841b47 Minor adjustments/comments to wal append_frames_vectored method 2025-08-25 09:47:06 -04:00
PThorpe92
0239088718 Use new append_frames_vectored WAL method to flush pager cache and commit write tx 2025-08-25 09:47:06 -04:00
Pekka Enberg
3f5878243f Merge 'Remove unnecessary argument from Pager::end_tx()' from Nikita Sivukhin
No need to pass `disable` flag to the `end_tx` method as it has that
info from connection itself

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2777
2025-08-25 15:34:41 +03:00
Preston Thorpe
040ceba2d6 Merge 'WAL txn: fix reads from DB file' from Nikita Sivukhin
- Transaction which was started with max_frame = 0 and
max_frame_read_lock_index = 0 can write to the WAL and in this case it
needs to read data back from WAL and not the DB file.
- Without cache spilling its hard to reproduce this issue for the turso-
db now, but I found this issue with sync-engine which do weird stuff
with the WAL which "simulates" cache spilling behaviour to some extent.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2735
2025-08-25 08:34:17 -04:00
Nikita Sivukhin
f7ad55b680 remove unnecessary argument 2025-08-25 12:24:39 +04:00
Nikita Sivukhin
c62b87d9b6 read from database file only if max_frame_read_lock_index is 0 and max_frame > min_frame
- transaction which was started with max_frame = 0 and max_frame_read_lock_index = 0
  can write to the WAL and in this case it needs to read data back from WAL
- without cache spilling its hard to reproduce this issue for the turso-db now,
  but I stumbled into this issue with sync-engine which do weird stuff with the WAL
  which "simulates" cache spilling behaviour to some extent
2025-08-25 11:36:58 +04:00
Avinash Sajjanshetty
48ce2a4a3e Set encryption ctx when cipher and key are set 2025-08-25 02:28:57 +05:30
Avinash Sajjanshetty
a4b9c33b81 Use the new API to init cipher 2025-08-24 16:15:13 +05:30
Pekka Enberg
22c9cb6618 s/PerConnEncryptionContext/EncryptionContext/ 2025-08-24 08:17:20 +03:00
Pekka Enberg
1b89273f10 Merge 'refactor encryption module and make it configurable' from Avinash Sajjanshetty
Previously, the encryption module had hardcoded a lot of things. This
refactor makes it slightly nice and makes it configurable.
Right now cipher algorithm is assumed and hardcoded, I will make that
configurable in the upcoming PR

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2722
2025-08-24 08:16:28 +03:00
PThorpe92
9a418f1d3e Replace a couple refcells with cell in pager 2025-08-23 15:55:01 -04:00
Pekka Enberg
156352fabc Merge ' should not return a Completion when there is a page cache hit' from Pedro Muniz
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2723
2025-08-22 07:33:28 +03:00
pedrocarlo
6b0ed08465 read_page should return No Completion when have a page cache hit 2025-08-21 14:39:24 -03:00
Avinash Sajjanshetty
3090545167 use encryption ctx instead of encryption key 2025-08-21 22:36:32 +05:30
Nikita Sivukhin
94f2e5a4b9 add truncate method to the page cache 2025-08-21 18:27:54 +04:00
Jussi Saurio
05f4cec34d Merge 'Wal api checkpoint seq' from Nikita Sivukhin
This PR adds information about checkpoint sequence number to the WAL raw
API. Will be used in the sync engine.
Depends on the #2699

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2707
2025-08-21 17:05:45 +03:00
Jussi Saurio
4ceadcca78 Merge 'Remove hardcoded flag usage in DBHeader for encryption' from Avinash Sajjanshetty
Previously, we just hardcoded the reserved space with encryption flag.
This patch removes that and sets the reserved space if a key was
specified during a creation of db

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2706
2025-08-21 15:46:41 +03:00
Nikita Sivukhin
69c39d5d8c replace wal_frames_count with wal_state method which return both frames count and checkpoint sequence 2025-08-21 15:13:23 +04:00
Avinash Sajjanshetty
1f93e77828 Remove hardcoded flag usage in DBHeader for encryption
Previously, we just hardcoded the reserved space with encryption flag.
This patch removes that and sets the reserved space if a key was
specified during a creation of db
2025-08-21 16:21:35 +05:30
Nikita Sivukhin
05931f70ce add optional upper_bound_inclusive parameter to some checkpoint modes
- will be used in sync-engine protocol
2025-08-21 14:12:11 +04:00
PThorpe92
4a2da6c262 Remove assertion for checkpoint seq in favor of selectively using cached pages 2025-08-20 18:26:55 -04:00
PThorpe92
7082086061 Remove ENV var and enable cache by default, track which pages were cached 2025-08-20 17:42:17 -04:00
PThorpe92
51e4cd0f1d Add debug assertion for cached pages used during checkpoint 2025-08-20 17:35:59 -04:00
PThorpe92
e28a38abc5 Fix wal tag safety issues, and add debug assertion that we are reading the proper frames 2025-08-20 17:28:48 -04:00
PThorpe92
d6d72d2966 Update Page to carry epoch of frame + checkpont seq to ensure proper cached page for chkpt 2025-08-20 17:28:17 -04:00
PThorpe92
fe7a5e98b8 Track frame_ids on PageInner and use the page cache for reading pages to checkpoint 2025-08-20 17:24:10 -04:00
Avinash Sajjanshetty
40a209c000 simplify feature flag usage for encryption 2025-08-20 12:49:38 +05:30
Avinash Sajjanshetty
bd9b4bbfd2 encrypt/decrypt when writing/reading from DB 2025-08-20 11:47:23 +05:30
Avinash Sajjanshetty
a6e9237c94 Set encryption key in pager and WAL 2025-08-20 11:39:09 +05:30
pedrocarlo
de1811dea7 abort completions on error 2025-08-19 10:48:21 -03:00
Jussi Saurio
ac37e89fe1 remove unused PAGE_ERROR flag 2025-08-19 15:44:16 +03:00
Jussi Saurio
7f1eac9560 Do not start or end transaction in nested statement 2025-08-19 13:03:14 +03:00
Jussi Saurio
9deedd917f Merge 'FaultyQuery enabled by default' from Pedro Muniz
FaultyQuery was disabled. We were not simulating faults.

Reviewed-by: Avinash Sajjanshetty (@avinassh)
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2594
2025-08-18 22:53:37 +03:00
pedrocarlo
aa7f8fbfc4 Tripwire for Wal Syncing + set syncing to true before calling sync 2025-08-18 15:34:20 -03:00
pedrocarlo
c5dc26f542 reset more pager state machines in rollback 2025-08-18 12:01:10 -03:00
Pere Diaz Bou
94cd504d4c core/mvcc: support schema_did change on commit_txn
This not only changes schema_did_change on commit_txn for mvcc, but also
extracts the connection transaction state from non mvcc transactions to
mvcc too.
2025-08-18 15:52:10 +02:00
PThorpe92
2c526c4c37 Add io_yield_x macros to reduce boilerplate 2025-08-16 16:14:00 -04:00
PThorpe92
2ad479f413 dont forget other instance where db file is expected size 2025-08-14 19:56:23 -04:00
PThorpe92
9017acd622 Ensure we fsync the db file in all paths that we checkpoint 2025-08-14 19:23:00 -04:00
Jussi Saurio
c75e4c1092 Fix non-4096 page sizes by making WAL header lazy 2025-08-14 12:40:58 +03:00