WAL API shouldn't be exposed by default because this is relatively
dangerous API which we use internally and ordinary users shouldn't not
be interested in it.
Reviewed-by: Pekka Enberg <penberg@iki.fi>
Closes#2424
We have some kind of transaction-local hack (`start_pages_in_frames`) for bookkeeping
how many pages are currently in the in-memory WAL frame cache,
I assume for performance reasons or whatever.
`wal.rollback()` clears all the frames from `shared.frame_cache` that the rollbacking tx is
allowed to clear, and then truncates `shared.pages_in_frames` to however much its local
`start_pages_in_frames` value was.
In `complete_append_frame`, we check if `frame_cache` has that key (page) already, and if not,
we add it to `pages_in_frames`.
However, `wal.rollback()` never _removes_ the key (page) if its value is empty, so we can end
up in a scenario where the `frame_cache` key for `page P` exists but has no frames, and so `page P`
does not get added to `pages_in_frames` in `complete_append_frame`.
This leads to a checkpoint data loss scenario:
- transaction rolls back, has start_pages_in_frames=0, so truncates
shared pages_in_frames to an empty vec. let's say `page P` key in `frame_cache` still remains
but it has no frames.
- The next time someone commits a frame for `page P`, it does NOT get added to `pages_in_frames`
because `frame_cache` has that key
- At some point, a PASSIVE checkpoint checkpoints `n` frames, but since `pages_in_frames` does not have
`page P`, it doesn't actually checkpoint it and all the "checkpointed" frames are simply thrown away
- very similar to the scenario in #2366
Remove the `start_pages_in_frames` hack entirely and just make `pages_in_frames` effectively
the same as `frame_cache.keys`. I think we could also just get rid of `pages_in_frames` and just use
`frame_cache.contains_key(p)` but maybe Pere can chime in here
## Background
When we get a new rowid using `op_new_rowid()`, we move to the end of
the btree to look at what the maximum rowid currently is, and then
increment it by one.
This requires a btree seek.
## Problem
If we were already on the rightmost page, this is a lot of unnecessary
work, including potentially a few page reads from disk (although to be
fair the ancestor pages are very likely to be in cache at this point.)
## Fix
Cache the rightmost page id whenever we enter it in
`move_to_rightmost()`, and invalidate it whenever we do a balancing
operation.
## Local benchmark results
```sql
Insert rows in batches/limbo_insert_1_rows
time: [23.333 µs 27.718 µs 35.801 µs]
change: [-7.7924% +0.8805% +12.841%] (p = 0.91 > 0.05)
No change in performance detected.
Insert rows in batches/limbo_insert_10_rows
time: [38.204 µs 38.381 µs 38.568 µs]
change: [-8.7188% -7.4786% -6.1955%] (p = 0.00 < 0.05)
Performance has improved.
Insert rows in batches/limbo_insert_100_rows
time: [158.39 µs 165.06 µs 178.37 µs]
change: [-21.000% -18.789% -15.666%] (p = 0.00 < 0.05)
Performance has improved.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2409
This should be safe to do as:
1. page cache is private per connection
2. since this connection wrote the flushed pages/frames, they are up to
date from its perspective
3. multiple concurrent statements inside one connection are not
snapshot-transactional even in sqlite
Reviewed-by: Pekka Enberg <penberg@iki.fi>
Closes#2407
https://github.com/tursodatabase/turso/pull/1256 switched cargo-dist to
Astral's forked version, but, recently, the official repository got a
new maintainer and started to be maintained again.
Their latest release, [v0.29.0](https://github.com/axodotdev/cargo-
dist/releases/tag/v0.29.0), now includes the features originally added
to Astral's version. So, probably it's a good time to switch back to the
official cargo-dist. That said, as there's no significant changes from
Astral's version, it's also fine to hold the current one.
Closes#2398
While working on #2151 I saw myself forced to do things like:
```rust
assert_eq!(
6,
*result
.next()
.await?
.unwrap()
.get_value(0)?
.as_integer()
.unwrap()
);
```
Just to get a simple value from a row, now with this PR users can just
do:
```rust
assert_eq!(6, result.get::<i32>(0)?);
```
(Thanks libsql devs, this is so much better!)
Closes#2377
This will save some work when yielding to IO. Previously, on every
invocation, if the record was a packed record, we parsed it and iterated
through the values to check for nulls. Now, the pre-seeking work is done
only once.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2394
Closes #2077-
This PR fixes an integer overflow bug that causes results from Turso to
differ from SQLite.
-Commit 1: Fixes incorrect logic that failed to detect integer overflow
when parsing long numeric strings.
-Commit 2: Handles the case where a parsed numeric string is stored as a
float and classified as PureInteger, but lies outside the integer range.
Previously, `parsed_value.as_integer()` would return None, causing Turso
to fall back to text comparison against numeric values. This caused
**another** erroneous result, as shown below.
`$> SELECT (-104614899632619 || 45597) > CAST(0 AS NUMERIC); -- tursodb
= 1(wrong), sqlite = 0`
Now, if Turso fails to convert a very long numeric string to an
integer, it tries to convert it to a float. This is in line with the
`static void applyNumericAffinity` function in sqlite3
**Before**
<img width="623" height="238" alt="Screenshot 2025-08-01 at 12 11 49 PM"
src="https://github.com/user-attachments/assets/796d6ff6-768b-40ef-
ac83-e0c55fff6bd9" />
**After**
`SELECT (104614899632619 || 45597) > CAST(0 AS NUMERIC); -- tursodb = 1,
sqlite = 1`
`SELECT (-104614899632619 || 45597) > CAST(0 AS NUMERIC); -- tursodb =
0, sqlite = 0`
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2397
