`FaultyQuery` causes frequent false positives in simulator due to the
following chain of events:
- we write rows and flush wal to disk
- inject fault during fsync which fails
- error is returned to caller, simulator thinks those rows dont exist
because the query failed
- we reopen the database i.e. read the WAL back to memory from disk, it
has those extra rows we think we didn't write
- assertion fails because table has more rows than simulator expected
More discussion about fsync behavior in issue #2091Closes#2110
`FaultyQuery` causes frequent false positives in simulator due to
the following chain of events:
- we write rows and flush wal to disk
- inject fault during fsync which fails
- error is returned to caller, simulator thinks those rows dont exist because the query failed
- we reopen the database i.e. read the WAL back to memory from disk, it has those extra rows we think we didn't write
- assertion fails because table has more rows than simulator expected
More discussion about fsync behavior in issue #2091
- Add generation for UNION/JOIN
- Rearchitect the oracle calling conventions to simplify the code paths
- Add brute force shrinking option by @echoumcp1
Closes#2049
as nilskch points out in #1807, Rust 1.88.0 is stricter about alignment
checks.
because rust integers default to `i32`, we were casting a pointer to an
`i32` as a pointer to an `i64` causing a panic when dereferenced due to
misalignment as rust expects it to be 8 byte aligned.
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#2064
- Fix not being able to create table while importing
* The behavior now aligns with SQLite so that if the table already
exists, all the rows are treated as data. If the table doesn't exist,
the first row is treated as the header from which column names for the
new table are populated.
- Insert in batches instead of one at a time
This was a pretty quick vibecoding effort tbh :]
Closes#2079Closes#2094
Small refactoring to reduce confusion (I was caught in this trap and set
`amount` to one in CDC branch during development)
Also, this PR slightly fix broken `concat_ws` emit logic.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2100
This PR unify the concept of a result that either have something done or
yields to IO, into a single type.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2103
- remove assumptions that record header size fits into 1 byte or serial
type fits into 1 byte
- add tests for record header size calculation
```sql
turso> CREATE TABLE t(x TEXT, y);
CREATE INDEX t_idx ON t(x);
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'a', 1); -- 1000 bytes of 'a'
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'b', 2);
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'c', 3);
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'd', 4);
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'e', 5);
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'f', 6);
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'g', 7);
INSERT INTO t VALUES (replace(zeroblob(1000), x'00', 'a') || 'h', 8);
SELECT COUNT(*) FROM t WHERE x >= replace(hex(zeroblob(100)), '00', 'a');
┌───────────┐
│ COUNT (*) │
├───────────┤
│ 8 │
└───────────┘
```
Fixes#2096Fixes#2088
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2098
## Background
PR #2065 fixed a bug with table btree seeks concerning boundaries of
leaf pages.
The issue was that if we were e.g. looking for the first key greater
than (GT) 100, we always assumed the key would either be found on the
left child page of a given divider (e.g. divider 102) or not at all,
which is incorrect. #2065 has more discussion and documentation about
this, so read that one for more context.
## This PR
We already had similar handling for index btrees as #2065 introduced for
table btrees, but it was baked into the `BTreeCursor` struct's seek
handling itself, whereas #2065 handled this on the VDBE side.
This PR unifies this handling for both table and index btrees by always
doing the additional cursor advancement in the VDBE.
Unfortunately, unlike table btrees, index btrees may also need to do an
additional advance when they are looking for an exact match. This
resulted in a bigger refactor than anticipated, since there are quite a
few VDBE instructions that may perform a seek, e.g.: `IdxInsert`,
`IdxDelete`, `Found`, `NotFound`, `NoConflict`. All of these can
potentially end up in a similar situation where the cursor needs one
more advance after the initial seek, and they were currently calling
`cursor.seek()` directly and expecting the `BTreeCursor` to handle the
auto-advance fallback internally.
For this reason, I have 1. removed the "TryAdvance"-ish logic from the
index btree internals and 2. extracted a common VDBE helper `fn
seek_internal()` - heavily based on the existing `op_seek_internal()`,
but decoupled from instructions and the program counter - which all the
interested VDBE instructions will call to delegate their seek logic.
Closes#2083
Reviewed-by: Nikita Sivukhin (@sivukhin)
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2084
- Fix not being able to create table while importing
* The behavior now aligns with SQLite so that if the table already
exists, all the rows are treated as data. If the table doesn't exist,
the first row is treated as the header from which column names for the
new table are populated.
- Insert in batches instead of one at a time
First step toward resolving
https://github.com/tursodatabase/limbo/issues/1643.
### This PR
With this change, the following two queries are considered equivalent:
```sql
SELECT value FROM generate_series(5, 50);
SELECT value FROM generate_series WHERE start = 5 AND stop = 50;
```
Arguments passed in parentheses to the virtual table name are now
matched to hidden columns.
Additionally, I fixed two bugs related to virtual tables.
### TODO (I'll handle this in a separate PR)
Column references are still not supported as table-valued function
arguments. The only difference is that previously, a query like:
```sql
SELECT one.value, series.value
FROM (SELECT 1 AS value) one, generate_series(one.value, 3) series;
```
would cause a panic. Now, it returns a proper error message instead.
Adding support for column references is more nuanced for two main
reasons:
* We need to ensure that in joins where a TVF depends on other tables,
those other tables are processed first. For example, in:
```sql
SELECT one.value, series.value
FROM generate_series(one.value, 3) series, (SELECT 1 AS value) one;
```
the one table must be processed by the top-level loop, and series must
be nested.
* For outer joins involving TVFs, the arguments must be treated as `ON`
predicates, not `WHERE` predicates.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1727
Closes: #1379
This PR consists of three main changes:
1. Rebuild `Pager` to set the correct page size for `buffer_pool`, `wal`
and other components when the database is uninitialized.
2. Persist the latest page size when allocate page 1.
3. Ensure all pragmas emit the correct transaction instructions,
preventing even a `page_size` read from triggering database
initialization.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2053
Biologically gopher is closer to beavers, than to hamsters, so it will
be much more correct to use beaver emoji.
And yes, if you merge this MR I would be proud of my contribution into
open source.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2067
PR #2065 fixed a bug with table btree seeks concerning boundaries
of leaf pages.
The issue was that if we were e.g. looking for the first key greater than
(GT) 100, we always assumed the key would either be found on the left child
page of a given divider (e.g. divider 102), which is incorrect. #2065 has more
discussion and documentation about this, so read that one for more context.
Anyway:
We already had similar handling for index btrees, but it was baked into
the `BTreeCursor` struct's seek handling itself, whereas #2065 handled this
on the VDBE side.
This PR unifies this handling for both table and index btrees by always doing
the additional cursor advancement in the VDBE.
Unfortunately, since indexes may also need to do an additional advance when they
are looking for an exact match, this resulted in a bigger refactor than anticipated,
since there are quite a few VDBE instructions that may perform a seek, e.g.:
`IdxInsert`, `IdxDelete`, `Found`, `NotFound`, `NoConflict`.
All of these can potentially end up in a similar situation where the cursor needs
one more advance after the initial seek.
For this reason, I have extracted a common VDBE helper `fn seek_internal()` which
all the interested VDBE instructions will call to delegate their seek logic.
The original `rowid_seek_fuzz` test had a design flaw: it inserted
contiguous non-random integers (so not really fuzzing), which prevented
issues such as the one fixed in #2065 from being discovered.
Further, the test has at some point also been neutered a bit by only
inserting 100 values which makes the btree very small, hiding
interactions between interior pages and neighboring leaf pages.
This should not be merged until #2065 is merged.
Closes#2081
