Commit Graph

1521 Commits

Author SHA1 Message Date
Preston Thorpe
8665d76c2e Merge 'Improve encryption module' from Avinash Sajjanshetty
This patch improves the encryption module:
1. Previously, we did not use the first 100 bytes in encryption. This
patch uses that portion as associated data, for protection against
tampering and corruption
2. Once the page 1 encrypted, on disk we store a special Turso header
(the first 16 bytes). During decryption we replace this with standard
SQLite's header (`"SQLite format 3\000"`). So that the upper layers (B
Tree or in Sync APIs) operate on the existing SQLite page expectations.
The format is:
```
///                    Turso Header (16 bytes)
///        ┌─────────┬───────┬────────┬──────────────────┐
///        │         │       │        │                  │
///        │  Turso  │Version│ Cipher │     Unused       │
///        │  (5)    │ (1)   │  (1)   │    (9 bytes)     │
///        │         │       │        │                  │
///        └─────────┴───────┴────────┴──────────────────┘
///         0-4      5       6        7-15
///
///        Standard SQLite Header: "SQLite format 3\0" (16 bytes)
///                            ↓
///        Turso Encrypted Header: "Turso" + Version + Cipher ID + Unused
```

Reviewed-by: Nikita Sivukhin (@sivukhin)
Reviewed-by: bit-aloo (@Shourya742)

Closes #3358
2025-09-29 11:04:31 -04:00
Pekka Enberg
f247b1a2bb core/storage: Wrap Pager::commit_info with RwLock
Also remove RefCells from CommitInfo because they're not only redundant,
but cause CommitInfo not to be Send.
2025-09-29 13:54:28 +03:00
Avinash Sajjanshetty
ec1bf8888c refactor to adress review comments 2025-09-28 22:03:47 +05:30
Pekka Enberg
d3abeb6281 core/storage: Wrap WalFile::{max,min}_frame with AtomicU64 2025-09-28 16:47:54 +03:00
Pekka Enberg
aba596441c core/storage: Wrap WalFile::max_frame_read_lock_index with AtomicUsize 2025-09-28 13:42:32 +03:00
Jussi Saurio
959165eed1 Merge 'core/storage: Mark Page as Send and Sync' from Pekka Enberg
Closes #3399
2025-09-28 08:08:46 +03:00
Avinash Sajjanshetty
c2453046fa clippy fixes 2025-09-27 18:16:51 +05:30
Avinash Sajjanshetty
a2df313ad5 Add documentation for the encryption module 2025-09-27 18:11:27 +05:30
Pekka Enberg
ce76aa11b2 core/storage: Mark Page as Send and Sync 2025-09-27 15:16:38 +03:00
Avinash Sajjanshetty
dc3d1fa36d Use the SQLite header as associated data for protection
against tampering and corruption.

Previously, we did not use the first 100 bytes in encryption
machinery. This patch changes that and uses that data as
associated data. So in case the header is corrupted or
tampered with, the decryption will fail.
2025-09-27 17:34:51 +05:30
Pekka Enberg
8d9d2dad1d core/storage: Wrap WalFile::syncing with AtomicBool 2025-09-27 14:07:26 +03:00
Pekka Enberg
931cf2658e core/storage: Display page category for rowid integrity check failure
Let's add more hints to hunt down the reason for #2896.
2025-09-26 18:25:49 +03:00
Pekka Enberg
60e9d1a1c4 core: Wrap Connection::is_nested_stmt in AtomicBool 2025-09-24 19:30:31 +03:00
Pekka Enberg
042a8dd031 core: Wrap Connection::wal_auto_checkpoint_disabled with AtomicBool 2025-09-24 09:12:46 +03:00
Pekka Enberg
aa95cb24ea core: Wrap Connection::page_size with AtomicU16 2025-09-24 09:12:46 +03:00
Pekka Enberg
f5d3962459 core: Wrap Connection::transaction_state with RwLock 2025-09-23 14:01:31 +03:00
Pekka Enberg
b94aa22499 core: Wrap Connection::schema in RwLock 2025-09-23 10:31:20 +03:00
Pekka Enberg
b857f94fe4 Merge 'core: Wrap Connection::pager in RwLock' from Pekka Enberg
Closes #3247
2025-09-23 07:29:09 +03:00
Pekka Enberg
aa454a6637 core: Wrap Connection::pager in RwLock 2025-09-22 17:02:08 +03:00
Jussi Saurio
4af49ef98c mvcc: simplify StateMachine
TransitionResult is an internal implementation detail that tells
an invocation of StateMachine::step() to continue looping, but it
is of no use to other callers.

For this reason, just return an IOResult from StateMachine::step()
which simplifies the result handling.
2025-09-22 16:37:31 +03:00
Preston Thorpe
44dc4c9636 Merge 'translate/emitter: Implement partial indexes' from Preston Thorpe
This PR adds support for partial indexes, e.g. `CREATE INDEX` with a
provided predicate
```sql
CREATE UNIQUE INDEX idx_expensive ON products(sku) where price > 100;
```
The PR does not yet implement support for using the partial indexes in
the optimizer.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3228
2025-09-22 09:09:54 -04:00
Pekka Enberg
372daef656 core: Wrap Pager::io_ctx in RwLock 2025-09-22 15:00:29 +03:00
Pekka Enberg
2af98223ae Merge 'Enable checksum tests if checksum feature is on' from Kacper Kołodziej
These tests fail if checksum feature is turned off.

Closes #3242
2025-09-22 14:40:34 +03:00
Kacper Kołodziej
76f2e4e217 Enable checksum tests if checksum feature is on
These tests fail if checksum feature is turned off.
2025-09-22 10:46:49 +02:00
Pekka Enberg
6f258b37d9 core/storage: Wrap Pager vacuum state in RwLock 2025-09-22 10:34:58 +03:00
Pekka Enberg
f053b76518 core/storage: Move vacuum state machines to VacuumState 2025-09-22 09:37:40 +03:00
Pekka Enberg
6280cfc59d Merge branch 'main' into sync-improvements 2025-09-22 07:35:39 +03:00
pedrocarlo
e5dfc942b1 remove some unnecessary unsafe impls 2025-09-21 13:29:59 -03:00
PThorpe92
21f6455190 Fix clippy warnings and tests 2025-09-20 14:38:50 -04:00
Pekka Enberg
9dda5a6263 Merge 'bugfix: clear reserved space for a reused page' from Avinash Sajjanshetty
fixes #3184

Closes #3198
2025-09-19 14:16:24 +03:00
Avinash Sajjanshetty
d5295fb45c Put the unused variable behind a flag as intended 2025-09-19 14:55:02 +05:30
Nikita Sivukhin
b106220743 main thread in browser can't execute parking - so we use parking lot in spin-lock style for that target 2025-09-19 13:21:00 +04:00
Pekka Enberg
508858dac6 core/storage: Wrap Pager::header_ref_state in RwLock 2025-09-19 08:38:45 +03:00
Samuel Marks
e333f151ba [*.rs] Resolve warnings (mostly "hiding a lifetime that's elided elsewhere is confusing") 2025-09-18 22:47:43 -05:00
Avinash Sajjanshetty
f0d705946c keep the reserved bytes check in debug_assertion flag 2025-09-18 22:37:16 +05:30
Pekka Enberg
a011548aac Merge 'core/storage: Wrap Pager::free_page_state with RwLock' from Pekka Enberg
Closes #3200
2025-09-18 17:17:53 +03:00
Pekka Enberg
beedb7a4aa Merge 'core: Rename Connection::_db to db' from Pekka Enberg
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #3196
2025-09-18 17:17:46 +03:00
Avinash Sajjanshetty
91e2a679b9 bugfix: clear reserved space for a reused page 2025-09-18 19:00:03 +05:30
Pekka Enberg
ecbe2c231f core: Rename Connection::_db to db 2025-09-18 16:00:41 +03:00
Pekka Enberg
7ac2e49a83 core/storage: Wrap Pager::free_page_state with RwLock 2025-09-18 15:46:23 +03:00
Pekka Enberg
d69cdd92d2 Merge 'core/storage: Switch Pager::max_page_count to AtomicU32' from Pekka Enberg
Closes #3197
2025-09-18 15:45:39 +03:00
Pekka Enberg
19513fd9a7 core/storage: Switch Pager::max_page_count to AtomicU32 2025-09-18 14:46:22 +03:00
Pekka Enberg
6e9684c763 Merge 'core/storage: Use AtomicU16 for Pager::reserved_space' from Pekka Enberg
Closes #3195
2025-09-18 14:46:08 +03:00
Preston Thorpe
ec79a9063d Merge 'remove io.blocks from btree balancing code' from Nikita Sivukhin
This PR removes `io.block` usage from B-Tree balancing code (similarly
as in the #3179)

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #3194
2025-09-18 07:24:51 -04:00
Pekka Enberg
6c48cb7043 Merge 'core: Use sequential consistency for atomics by default' from Pekka Enberg
We use relaxed ordering in a lot of places where we really need to
ensure all CPUs see the write. Switch to sequential consistency, unless
acquire/release is explicitly used. If there are places that can be
optimized, we can switch to relaxed case-by-case, but have a comment
explaning *why* it is safe.

Closes #3193
2025-09-18 14:19:54 +03:00
Pekka Enberg
50acebd88e core/storage: Use AtomicU16 for Pager::reserved_space 2025-09-18 14:19:13 +03:00
Pekka Enberg
8337e86794 core: Use sequential consistency for atomics by default
We use relaxed ordering in a lot of places where we really need to
ensure all CPUs see the write. Switch to sequential consistency, unless
acquire/release is explicitly used. If there are places that can be
optimized, we can switch to relaxed case-by-case, but have a comment
explaning *why* it is safe.
2025-09-18 13:38:13 +03:00
Nikita Sivukhin
1e0fb143f6 remove io.blocks from btree balancing code 2025-09-18 14:28:53 +04:00
Pekka Enberg
2a5284afb9 core/storage: Use AtomicU32 for Pager::page_size 2025-09-18 11:33:32 +03:00
Pekka Enberg
e6d994dee0 core/storage: Wrap Pager::allocate_page_state with RwLock 2025-09-18 10:55:55 +03:00