Commit Graph

4260 Commits

Author SHA1 Message Date
Preston Thorpe
7f8e181cda Merge 'Add documentation and rename functions' from Mikaël Francoeur
This PR adds documentation and renames some functions. Among other
things, I renamed everything that was still called `owned_value` to
either `db_value` or `value` (after
https://github.com/tursodatabase/turso/pull/1488).

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2619
2025-08-16 17:45:33 -04:00
PThorpe92
2c526c4c37 Add io_yield_x macros to reduce boilerplate 2025-08-16 16:14:00 -04:00
Pekka Enberg
39fe7a0a32 Merge 'core/printf: support for the %i operand' from Luiz Gustavo
This continues the work for `printf` support from from tthe issue
https://github.com/tursodatabase/turso/issues/885.
This PR adds support for `%i` operands in the printf function. An
example of a command that works now and didn't before:
```sql
SELECT PRINTF("Decimals: %d %i", 200, 300);
```
This is Just an initial very simple contribution to get my feet wet and
learn the basics about the project and how contributions here work. I
intend to do something more meaningful in next contributions. Probably
will try to finish support for missing `printf` functionality also.

Closes #2615
2025-08-16 09:53:10 +03:00
Mikaël Francoeur
2ee0132afe rename functions 2025-08-15 17:08:53 -04:00
Jussi Saurio
c03517da40 Merge 'Fix distinct order by' from Jussi Saurio
Closes #2612
Two commits:
## Simplify ORDER BY sorter column remapping
In case an ORDER BY column exactly matches a result column in the
SELECT,
the insertion of the result column into the ORDER BY sorter can be
skipped
because it's already necessarily inserted as a sorting column.
For this reason we have a mapping to know what index a given result
column
has in the order by sorter.
This commit makes that mapping much simpler.
## Fix DISTINCT with ORDER BY
We had a bug where we were checking for duplicates in the DISTINCT
index based on both the result column count plus any ORDER BY columns
not present in the DISTINCT clause.
This is wrong, so fix it by only using the result columns for the
dedupe check.

Closes #2614
2025-08-15 23:43:29 +03:00
Mikaël Francoeur
4a986080d2 add documentation 2025-08-15 14:59:37 -04:00
Jussi Saurio
d2cfe06aa5 Fix DISTINCT with ORDER BY
We had a bug where we were checking for duplicates in the DISTINCT
index based on both the result column count plus any ORDER BY columns
not present in the DISTINCT clause.

This is wrong, so fix it by only using the result columns for the
dedupe check.
2025-08-15 15:49:55 +03:00
luizgfc
e370b714c9 core/printf: support for the %i operand 2025-08-15 09:48:55 -03:00
Jussi Saurio
a99c8a8ca0 Simplify ORDER BY sorter column remapping
In case an ORDER BY column exactly matches a result column in the SELECT,
the insertion of the result column into the ORDER BY sorter can be skipped
because it's already necessarily inserted as a sorting column.

For this reason we have a mapping to know what index a given result column
has in the order by sorter.

This commit makes that mapping much simpler.
2025-08-15 15:48:41 +03:00
Pekka Enberg
c1c2b45141 core/vdbe: Drop excessive logging 2025-08-15 14:56:06 +03:00
Nikita Sivukhin
71bbc36f61 adjust log message 2025-08-15 13:19:46 +04:00
Nikita Sivukhin
2361587d81 cargo fmt 2025-08-15 13:18:50 +04:00
Nikita Sivukhin
9b08f8c0f2 set in-memory WAL information according to last commited frame
- before that pages_in_frames, frame_cache and last_checksum was set to the latest written (not commited!) frame in the WAL found on disk
2025-08-15 13:13:33 +04:00
Pekka Enberg
e4f424f9e6 Merge 'Properly implement CLI command' from Preston Thorpe
Closes #2588
SQLite internally implements `.clone` by doing something like piping
`.dump` into a new connection to a database attached to the file you
want. This PR implements that by adding an `ApplyWriter` that implements
`std::fmt::Write`, and refactors our current `.dump` plumbing to work
with any `Write` interface, so we can `.dump` to stdout or `.dump` to a
new connection, therefore cloning the database.

Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2590
2025-08-15 10:44:57 +03:00
PThorpe92
7c021c01f8 Remove copy_db method from connection 2025-08-14 21:31:13 -04:00
PThorpe92
cc2fed3297 Remove copy_to API from file IO trait 2025-08-14 21:31:13 -04:00
PThorpe92
285dcdd2c1 Prevent potential corruption from copying db file without holding proper locks 2025-08-14 21:31:13 -04:00
PThorpe92
3c088dda59 Update callsites of copy_to Database impl 2025-08-14 21:31:13 -04:00
PThorpe92
55f09a01c4 Update copy_to method in file trait to separate source and destination IO 2025-08-14 21:31:13 -04:00
PThorpe92
2ad479f413 dont forget other instance where db file is expected size 2025-08-14 19:56:23 -04:00
PThorpe92
9017acd622 Ensure we fsync the db file in all paths that we checkpoint 2025-08-14 19:23:00 -04:00
Preston Thorpe
5cea0f572e Merge 'Revive async io extension PR' from Preston Thorpe
bringing #1127 back to life, except better because this doesn't add
Tokio as a dependency for extension lib just for tests.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2418
2025-08-14 16:10:09 -04:00
Preston Thorpe
e9d23e7fea apply copilot suggestion
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-14 16:08:55 -04:00
Preston Thorpe
a425c72ddc Merge 'Add assertion for expected write amount in writev callback' from Preston Thorpe
as @avinassh pointed out, if we have an error in a `writev` call, it
could go unnoticed and the checkpoint could proceed as expected after
not writing the expected total amt.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2601
2025-08-14 14:05:08 -04:00
PThorpe92
5a7c9325e2 Add assertion for expected write amount in writev callback 2025-08-14 10:53:16 -04:00
Jussi Saurio
5da76c9125 Allow index in UPDATE for point queries (i.e. max 1 row affected) 2025-08-14 15:58:01 +03:00
Jussi Saurio
cd3b4bccd3 Fix UPDATE: Do not use an index for iteration if that index is going to be updated
Closes #2598
2025-08-14 15:35:00 +03:00
Jussi Saurio
bd8c6f3c7c make PageSize more robust: only accept literal '1' value if it comes directly from db header 2025-08-14 12:40:58 +03:00
Jussi Saurio
c2e89f94f8 Change more page size panics to corrupt errors 2025-08-14 12:40:58 +03:00
Jussi Saurio
f94fa2bbbe salt tweak 2025-08-14 12:40:58 +03:00
Jussi Saurio
38bb0719cc read from disk tweak 2025-08-14 12:40:58 +03:00
Jussi Saurio
a2a88e2c69 Make exception for page size literal value 1 2025-08-14 12:40:58 +03:00
Jussi Saurio
c75e4c1092 Fix non-4096 page sizes by making WAL header lazy 2025-08-14 12:40:58 +03:00
Jussi Saurio
f8620a9869 Use non-hardcoded size for BTreeCursor immutablerecord 2025-08-14 12:40:58 +03:00
Jussi Saurio
f5e27f23ad Use type-safe PageSize newtype for connection.page_size 2025-08-14 12:40:58 +03:00
Jussi Saurio
bb21bd93da Use type-safe PageSize newtype for pager.page_size 2025-08-14 12:40:58 +03:00
Jussi Saurio
ee58b7bd86 Add fn read_header() to DatabaseStorage trait 2025-08-14 12:40:58 +03:00
Jussi Saurio
a2a6feb193 Merge 'Use BufferPool owned by Database instead of a static global' from Jussi Saurio
## Problem
There are several problems with our current statically allocated
`BufferPool`.
1. You cannot open two databases in the same process with different page
sizes, because the `BufferPool`'s `Arena`s will be locked forever into
the page size of the first database. This is the case regardless of
whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.
2. It is impossible to even write Rust tests for different page sizes
because of this, assuming the test uses a single process.
## Solution
Make `Database` own `BufferPool` instead of it being statically
allocated, so this problem goes away.
Note that I didn't touch the still statically-allocated
`TEMP_BUFFER_CACHE`, because it should continue to work regardless of
this change. It should only be a problem if the user has two or more
databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a
time, so the rest of the allocations will go through the global
allocator instead.
## Notes
I extracted this change out from #2569, because I didn't want it to be
smuggled in without being reviewed as an individual piece.

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2596
2025-08-14 12:40:32 +03:00
Jussi Saurio
69d8a73028 Merge 'use virtual root page for sqlite_schema' from Mikaël Francoeur
This PR fixes a problem where `sqlite_schema` could be read before page
1 was allocated.
The fix is similar to that in SQLite. In SQLite, if `btreeCursor()` sees
that the root page is 1 and that the b-tree is empty, it sets the page
to 0 ([here](https://github.com/sqlite/sqlite/blob/master/src/btree.c#L4
691-L4696)). SQLite's `moveToRoot()` then uses this special value to
return `CURSOR_INVALID` with no rows ([here](https://github.com/sqlite/s
qlite/blob/master/src/btree.c#L5538-L5540)).
Fixes https://github.com/tursodatabase/turso/issues/2449

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2551
2025-08-14 11:08:11 +03:00
Jussi Saurio
d7186c7d7b Merge 'Add support for unlikely(X)' from bit-aloo
Implements the unlikely(X) function. Removes runtime implementations of
likely(), unlikely() and likelihood(), replacing them with panics if
they reach the VDBE.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2559
2025-08-14 10:56:27 +03:00
Jussi Saurio
78f1ed979e Merge 'io_uring: Gracefully handle submission queue overflow' from Preston Thorpe
Current handling is not ideal, this adds proper squeue overflow handling
by ensuring everything is still submitted in-order

Closes #2586
2025-08-14 10:55:17 +03:00
Jussi Saurio
0b17957f4e Merge 'Implement normal views' from Glauber Costa
Now that we actually implemented the statement parsing around views,
implementing normal SQLite views is relatively trivial, as they are just
an alias to a query.
We'll implement them now to get them out of the way, and then I'll go
back to DBSP

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2591
2025-08-14 10:54:46 +03:00
Jussi Saurio
359cba0474 Use BufferPool owned by Database instead of a static global
Problem

There are several problems with our current statically allocated
`BufferPool`.

1. You cannot open two databases in the same process with different
page sizes, because the `BufferPool`'s `Arena`s will be locked forever
into the page size of the first database. This is the case regardless
of whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.

2. It is impossible to even write Rust tests for different page sizes because
of this, assuming the test uses a single process.

Solution

Make `Database` own `BufferPool` instead of it being statically allocated, so this
problem goes away.

Note that I didn't touch the still statically-allocated `TEMP_BUFFER_CACHE`, because
it should continue to work regardless of this change. It should only be a problem if
the user has two or more databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a time, so the rest
of the allocations will go through the global allocator instead.

Notes

I extracted this change out from #2569, because I didn't want it to be smuggled in without
being reviewed as an individual piece.
2025-08-14 10:29:52 +03:00
bit-aloo
32e59614c7 remove unnecessary copy instr in likelihood, likely and unlikely 2025-08-14 09:08:32 +05:30
Preston Thorpe
be2c0ec6ab Merge 'Refactor: atomic ordering' from Preston Thorpe
Sequential is very rarely actually needed, we can very safely use
Acquire / Release for loads/stores, and some of these aren't guarding
anything and can use Relaxed.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2548
2025-08-13 22:39:50 -04:00
PThorpe92
3840708970 Add safety comment 2025-08-13 21:42:19 -04:00
PThorpe92
0e3750b85b Integrate changes to vfs IO impl for callbacks 2025-08-13 21:42:18 -04:00
Preston Thorpe
78abe72762 Merge 'fix: Handle fresh INSERTs in materialized view incremental maintenance' from Glauber Costa
The op_insert function was incorrectly trying to capture an "old record"
for fresh INSERT operations when a table had dependent materialized
views. This caused a "Cannot delete: no current row" error because the
cursor wasn't positioned on any row for new inserts.
The issue was introduced in commit f38333b3 which refactored the state
machine for incremental view handling but didn't properly distinguish
between:
- Fresh INSERT operations (no old record exists)
- UPDATE operations without rowid change (old record should be captured)
- UPDATE operations with rowid change (already handled by DELETE)
This fix checks if cursor.rowid() returns a value before attempting to
capture the old record. If no row exists (fresh INSERT), we correctly
set old_record to None instead of erroring out.
I am also including tests to make sure this doesn't break. The reason I
didn't include tests earlier is that I didn't know it was possible to
run the tests under a flag. But in here, I am just adding the flag to
the execution script.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2579
2025-08-13 21:36:19 -04:00
Preston Thorpe
c3e29087a8 Merge 'Fix: do computations on usable_space as usize, not as u16' from Jussi Saurio
Otherwise page size 65536 will not work as casting to u16 will make it
wrap around to 0.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2583
2025-08-13 17:08:43 -04:00
Mikaël Francoeur
07ef47924c use virtual root page for sqlite_schema 2025-08-13 16:31:21 -04:00