Commit Graph

1219 Commits

Author SHA1 Message Date
pedrocarlo
aa7f8fbfc4 Tripwire for Wal Syncing + set syncing to true before calling sync 2025-08-18 15:34:20 -03:00
pedrocarlo
c5dc26f542 reset more pager state machines in rollback 2025-08-18 12:01:10 -03:00
Jussi Saurio
9abc63d853 Add a bit of abstraction for creating EQ cursorcontexts 2025-08-18 13:13:02 +03:00
Jussi Saurio
3eb89982ba Remove obsolete FIXME 2025-08-18 12:08:40 +03:00
Jussi Saurio
50fd7ec58b Refactor: use regular save/restore context mechanism for delete balancing
- Removes special `DeleteSavepoint` and uses the existing cursor restoration
  mechanism.
- This required some restructuring of `DeleteState` to avoid cloning it, i.e.
  some negotations with the borrow checker.
- CursorContext now takes a SeekOp as well to allow retaining the behavior
  that we use LT for seeking after a delete-induced rebalancing. This behavior
  will probably be removed as part of fixing #2004, but here I am trying to
  preserve the current semantics.
2025-08-18 11:58:00 +03:00
PThorpe92
605b3e233f Move in memory reset logic to before truncation in wal restart 2025-08-16 17:42:22 -04:00
PThorpe92
eae68f3669 Update stale in memory wal header after restarting log 2025-08-16 17:36:48 -04:00
PThorpe92
2c526c4c37 Add io_yield_x macros to reduce boilerplate 2025-08-16 16:14:00 -04:00
Nikita Sivukhin
71bbc36f61 adjust log message 2025-08-15 13:19:46 +04:00
Nikita Sivukhin
2361587d81 cargo fmt 2025-08-15 13:18:50 +04:00
Nikita Sivukhin
9b08f8c0f2 set in-memory WAL information according to last commited frame
- before that pages_in_frames, frame_cache and last_checksum was set to the latest written (not commited!) frame in the WAL found on disk
2025-08-15 13:13:33 +04:00
Pekka Enberg
e4f424f9e6 Merge 'Properly implement CLI command' from Preston Thorpe
Closes #2588
SQLite internally implements `.clone` by doing something like piping
`.dump` into a new connection to a database attached to the file you
want. This PR implements that by adding an `ApplyWriter` that implements
`std::fmt::Write`, and refactors our current `.dump` plumbing to work
with any `Write` interface, so we can `.dump` to stdout or `.dump` to a
new connection, therefore cloning the database.

Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2590
2025-08-15 10:44:57 +03:00
PThorpe92
cc2fed3297 Remove copy_to API from file IO trait 2025-08-14 21:31:13 -04:00
PThorpe92
3c088dda59 Update callsites of copy_to Database impl 2025-08-14 21:31:13 -04:00
PThorpe92
2ad479f413 dont forget other instance where db file is expected size 2025-08-14 19:56:23 -04:00
PThorpe92
9017acd622 Ensure we fsync the db file in all paths that we checkpoint 2025-08-14 19:23:00 -04:00
PThorpe92
5a7c9325e2 Add assertion for expected write amount in writev callback 2025-08-14 10:53:16 -04:00
Jussi Saurio
bd8c6f3c7c make PageSize more robust: only accept literal '1' value if it comes directly from db header 2025-08-14 12:40:58 +03:00
Jussi Saurio
c2e89f94f8 Change more page size panics to corrupt errors 2025-08-14 12:40:58 +03:00
Jussi Saurio
f94fa2bbbe salt tweak 2025-08-14 12:40:58 +03:00
Jussi Saurio
a2a88e2c69 Make exception for page size literal value 1 2025-08-14 12:40:58 +03:00
Jussi Saurio
c75e4c1092 Fix non-4096 page sizes by making WAL header lazy 2025-08-14 12:40:58 +03:00
Jussi Saurio
f8620a9869 Use non-hardcoded size for BTreeCursor immutablerecord 2025-08-14 12:40:58 +03:00
Jussi Saurio
bb21bd93da Use type-safe PageSize newtype for pager.page_size 2025-08-14 12:40:58 +03:00
Jussi Saurio
ee58b7bd86 Add fn read_header() to DatabaseStorage trait 2025-08-14 12:40:58 +03:00
Jussi Saurio
a2a6feb193 Merge 'Use BufferPool owned by Database instead of a static global' from Jussi Saurio
## Problem
There are several problems with our current statically allocated
`BufferPool`.
1. You cannot open two databases in the same process with different page
sizes, because the `BufferPool`'s `Arena`s will be locked forever into
the page size of the first database. This is the case regardless of
whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.
2. It is impossible to even write Rust tests for different page sizes
because of this, assuming the test uses a single process.
## Solution
Make `Database` own `BufferPool` instead of it being statically
allocated, so this problem goes away.
Note that I didn't touch the still statically-allocated
`TEMP_BUFFER_CACHE`, because it should continue to work regardless of
this change. It should only be a problem if the user has two or more
databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a
time, so the rest of the allocations will go through the global
allocator instead.
## Notes
I extracted this change out from #2569, because I didn't want it to be
smuggled in without being reviewed as an individual piece.

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2596
2025-08-14 12:40:32 +03:00
Jussi Saurio
69d8a73028 Merge 'use virtual root page for sqlite_schema' from Mikaël Francoeur
This PR fixes a problem where `sqlite_schema` could be read before page
1 was allocated.
The fix is similar to that in SQLite. In SQLite, if `btreeCursor()` sees
that the root page is 1 and that the b-tree is empty, it sets the page
to 0 ([here](https://github.com/sqlite/sqlite/blob/master/src/btree.c#L4
691-L4696)). SQLite's `moveToRoot()` then uses this special value to
return `CURSOR_INVALID` with no rows ([here](https://github.com/sqlite/s
qlite/blob/master/src/btree.c#L5538-L5540)).
Fixes https://github.com/tursodatabase/turso/issues/2449

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2551
2025-08-14 11:08:11 +03:00
Jussi Saurio
359cba0474 Use BufferPool owned by Database instead of a static global
Problem

There are several problems with our current statically allocated
`BufferPool`.

1. You cannot open two databases in the same process with different
page sizes, because the `BufferPool`'s `Arena`s will be locked forever
into the page size of the first database. This is the case regardless
of whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.

2. It is impossible to even write Rust tests for different page sizes because
of this, assuming the test uses a single process.

Solution

Make `Database` own `BufferPool` instead of it being statically allocated, so this
problem goes away.

Note that I didn't touch the still statically-allocated `TEMP_BUFFER_CACHE`, because
it should continue to work regardless of this change. It should only be a problem if
the user has two or more databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a time, so the rest
of the allocations will go through the global allocator instead.

Notes

I extracted this change out from #2569, because I didn't want it to be smuggled in without
being reviewed as an individual piece.
2025-08-14 10:29:52 +03:00
Preston Thorpe
be2c0ec6ab Merge 'Refactor: atomic ordering' from Preston Thorpe
Sequential is very rarely actually needed, we can very safely use
Acquire / Release for loads/stores, and some of these aren't guarding
anything and can use Relaxed.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2548
2025-08-13 22:39:50 -04:00
Preston Thorpe
c3e29087a8 Merge 'Fix: do computations on usable_space as usize, not as u16' from Jussi Saurio
Otherwise page size 65536 will not work as casting to u16 will make it
wrap around to 0.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2583
2025-08-13 17:08:43 -04:00
Mikaël Francoeur
07ef47924c use virtual root page for sqlite_schema 2025-08-13 16:31:21 -04:00
Jussi Saurio
fd72a2ff20 Fix: do computations on usable_space as usize, not as u16
Otherwise page size 65536 will not work as casting to u16 will make
it wrap around to 0.
2025-08-13 17:20:29 +03:00
PThorpe92
f1475bd5ac Remove bool return value from page set_locked 2025-08-13 10:17:33 -04:00
PThorpe92
614a0a45a6 Relax and fix memory ordering 2025-08-13 10:09:37 -04:00
Nikita Sivukhin
5838efe7dd rename flag to wal_auto_checkpoint_disabled 2025-08-13 15:26:25 +04:00
Nikita Sivukhin
f9f56f454c change semantic of wal_disable_checkpoint flag - it disable only automatic checkpoint (on shutdown or when WAL is grows too big) 2025-08-13 15:23:18 +04:00
Nikita Sivukhin
a78992e309 add assertion with more context 2025-08-13 15:23:18 +04:00
pedrocarlo
187ffb5698 adjust test 2025-08-13 10:24:55 +03:00
pedrocarlo
8d48be0e61 cleanup 2025-08-13 10:24:55 +03:00
pedrocarlo
c1975cdfa1 ptrmap functions should bubble completions 2025-08-13 10:24:55 +03:00
pedrocarlo
925d4b15e1 adjust more pager IO returns 2025-08-13 10:24:55 +03:00
pedrocarlo
56905fc340 refactor checkpoint in pager 2025-08-13 10:24:55 +03:00
pedrocarlo
a7f5912e7d adjust wal checkpoint to return completions 2025-08-13 10:24:55 +03:00
pedrocarlo
10ad43d9e0 sync should return completion 2025-08-13 10:24:55 +03:00
pedrocarlo
a7873d8907 remove write counters from Wal impls 2025-08-13 10:24:55 +03:00
pedrocarlo
494fdb828b buuble completions in pager 2025-08-13 10:24:55 +03:00
pedrocarlo
ccc22863c6 remove return_if_locked and return_if_locked_maybe_load 2025-08-13 10:24:55 +03:00
pedrocarlo
f95625a06c bubble completions in btree 2025-08-13 10:24:55 +03:00
pedrocarlo
2e68296107 create IOCompletions 2025-08-13 10:24:55 +03:00
Jussi Saurio
43628e2a9a Merge 'IO Cleanups to use and ' from Pedro Muniz
Depends on #2512 .
This is an attempt to remove all the noise from my IO completions
refactor into separate PRs

Closes #2566
2025-08-13 10:24:15 +03:00