Closes#2555
## Problem
The main problem we had with the current implementation of
`init_pager()` was that the WAL header was eagerly allocated and written
to disk with a page size, and a potential already-set page size on an
initialized database was not checked. Given this scenario:
- Initialized database with e.g. page size 512 but no WAL
- Tursodb connects to DB
It would not check the database file for the page size and instead would
initialize the WAL header with the default 4096 page size, plus
initialize the `BufferPool` similarly with the wrong size, and then
panic when reading pages from the DB, expecting to read `4096` instead
of `512`, as demonstrated in the reproduction of #2555.
## Fix
1. Add `Database::read_page_size_from_db_header()` method that can be
used in the above cases
2. Initialize the WAL header lazily during the first frame append, using
the existing `WalFile::ensure_header_if_needed()` method, removing the
need to eagerly pass `page_size` when constructing the in-memory
`WalFileShared` structure.
## Reader notes
This PR follows a fairly logical commit-by-commit structure so you'll
preferably want to read it that way.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2569
## Problem
There are several problems with our current statically allocated
`BufferPool`.
1. You cannot open two databases in the same process with different page
sizes, because the `BufferPool`'s `Arena`s will be locked forever into
the page size of the first database. This is the case regardless of
whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.
2. It is impossible to even write Rust tests for different page sizes
because of this, assuming the test uses a single process.
## Solution
Make `Database` own `BufferPool` instead of it being statically
allocated, so this problem goes away.
Note that I didn't touch the still statically-allocated
`TEMP_BUFFER_CACHE`, because it should continue to work regardless of
this change. It should only be a problem if the user has two or more
databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a
time, so the rest of the allocations will go through the global
allocator instead.
## Notes
I extracted this change out from #2569, because I didn't want it to be
smuggled in without being reviewed as an individual piece.
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#2596
This PR ignores changes from `TURSO_SYNC_TABLE_NAME` meta table in order
to not generate unnecessary push commands when nothing actually changed
on the client side.
Closes#2597
Implements the unlikely(X) function. Removes runtime implementations of
likely(), unlikely() and likelihood(), replacing them with panics if
they reach the VDBE.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2559
Now that we actually implemented the statement parsing around views,
implementing normal SQLite views is relatively trivial, as they are just
an alias to a query.
We'll implement them now to get them out of the way, and then I'll go
back to DBSP
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2591
Problem
There are several problems with our current statically allocated
`BufferPool`.
1. You cannot open two databases in the same process with different
page sizes, because the `BufferPool`'s `Arena`s will be locked forever
into the page size of the first database. This is the case regardless
of whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.
2. It is impossible to even write Rust tests for different page sizes because
of this, assuming the test uses a single process.
Solution
Make `Database` own `BufferPool` instead of it being statically allocated, so this
problem goes away.
Note that I didn't touch the still statically-allocated `TEMP_BUFFER_CACHE`, because
it should continue to work regardless of this change. It should only be a problem if
the user has two or more databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a time, so the rest
of the allocations will go through the global allocator instead.
Notes
I extracted this change out from #2569, because I didn't want it to be smuggled in without
being reviewed as an individual piece.
Sequential is very rarely actually needed, we can very safely use
Acquire / Release for loads/stores, and some of these aren't guarding
anything and can use Relaxed.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2548
The op_insert function was incorrectly trying to capture an "old record"
for fresh INSERT operations when a table had dependent materialized
views. This caused a "Cannot delete: no current row" error because the
cursor wasn't positioned on any row for new inserts.
The issue was introduced in commit f38333b3 which refactored the state
machine for incremental view handling but didn't properly distinguish
between:
- Fresh INSERT operations (no old record exists)
- UPDATE operations without rowid change (old record should be captured)
- UPDATE operations with rowid change (already handled by DELETE)
This fix checks if cursor.rowid() returns a value before attempting to
capture the old record. If no row exists (fresh INSERT), we correctly
set old_record to None instead of erroring out.
I am also including tests to make sure this doesn't break. The reason I
didn't include tests earlier is that I didn't know it was possible to
run the tests under a flag. But in here, I am just adding the flag to
the execution script.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2579
A lot of the structures we have - like the ones under Schema, are
specific for materialized views. In preparation to adding normal views,
rename them, so things are less confusing.
I modified the `docs/manual.md` file to store the current I/O model that
Turso follows. I want some feedback on the text and on the design as
well!
Closes#2308
This PR fixes several small bugs around the sync-engine:
1. WAL pull handled end of the "frame" iterator incorrectly - this is
fixed in the https://github.com/tursodatabase/turso/commit/eff8d8540d1e8
3214459822ac6eeb0d3409ecc24 and test with concurrent DBs were added
2. Using `:memory:` in the sync engine lead to weird behavior because
engine will create different `MemoryIO` instances but turso-core under
the hood will use global registry of databases. I **changed** criteria
for determining in-memory databases by checking the prefix of the path
to be equal to `:memory:` (https://github.com/tursodatabase/turso/commit
/80476b3069f2cd460f368e10b4b2ef51f7608077)
3. Switched from `Buffer` to `Vec<u8>` for now as browser seems to not
support `Buffer` natively: https://github.com/tursodatabase/turso/commit
/2ca8a15dcc8ec79eddfaf7068fab4e1aa3241506
4. Added tracing to the sync engine: https://github.com/tursodatabase/tu
rso/commit/33ef1aa0da7dfe9416025e26e98f9cf9d48c9119
Closes#2582
This PR change semantic of `wal_disable_checkpoint` flag and make it
disable only "automatic" checkpoint behaviour which at the moment
includes:
1. Checkpoint on shutdown
2. Checkpoint when WAL reach certain size
sync-engine needs checkpoint to be in full control but it also will
issue `TRUNCATE` checkpoint at some certain moments of time - so
complete disablement of checkpoints will not suffice.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2580
Indices enabled by default in the shell - so let's enable them also in
the SDK
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2581
