Commit Graph

8540 Commits

Author SHA1 Message Date
Jussi Saurio
cc46bfb6d8 add gen-bigass-database.py
ability to create an equivalent big schema so that the issues
reported by Preston using that larger schema can be reproduced
and fixed.
2025-09-08 13:05:33 -04:00
Preston Thorpe
8d05336522 Merge 'Fix affinity handling in MakeRecord' from Pekka Enberg
Closes #2966
2025-09-08 12:14:33 -04:00
Pekka Enberg
71a812ce55 Merge 'Fix infinite loop when IO failure happens on allocating first page' from Preston Thorpe
closes #2919

Reviewed-by: Pedro Muniz (@pedrocarlo)

Closes #2968
2025-09-08 18:59:34 +03:00
Pekka Enberg
86baa06600 tests/integration: Add affinity differential fuzz test 2025-09-08 18:59:20 +03:00
PThorpe92
237b9fefd7 Fix infinite loop when IO failure happens on allocating first page 2025-09-08 11:49:33 -04:00
Pekka Enberg
0c6398c935 core/vdbe: Fix apply_affinity_char() text parsing
We need strict parsing in apply_affinity_char() to avoid transforming
non-numeric values (for example, "1a") into numeric values.
2025-09-08 18:49:13 +03:00
Pekka Enberg
7f002548fd tests/integration: Enable min_max_agg_fuzz() test case 2025-09-08 18:49:13 +03:00
Pekka Enberg
f88f39082a core/vdbe: Fix MakeRecord affinity handling
The MakeRecord instruction now accepts an optional affinity_str
parameter that applies column-specific type conversions before creating
records. When provided, the affinity string is applied
character-by-character to each register using the existing
apply_affinity_char() function, matching SQLite's behavior.

Fixes #2040
Fixes #2041
2025-09-08 18:49:13 +03:00
Pekka Enberg
081a7b563b Merge 'Fix crash in Next opcode if cursor stack has no pages' from Jussi Saurio
Closes #2924
Unsure if this fix is that great, but it does fix the issue described in
#2924 -- added minimal regression test to illustrate the behavior
This crash requires a pretty specific set of circumstances:
- 3-way join with two innermost being left joins
- nullable seek key on the innermost table:
    * middle table gets nulled out because no matches with the outermost
table
    * hence when we seek the innermost table using middle table values,
the seek key is null, so `Insn::IsNull` entirely skips the innermost
table
Perhaps a bytecode plan illustrates this better:
```sql
turso> explain select a.x, b.x, c.x from a left join b on a.y=b.x left join c on b.y=c.x;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     34    0                    0   Start at 34
1     OpenRead           0     2     0                    0   table=a, root=2, iDb=0
2     OpenRead           1     4     0                    0   table=b, root=4, iDb=0
3     OpenRead           2     5     0                    0   index=sqlite_autoindex_b_1, root=5, iDb=0
4     OpenRead           3     7     0                    0   index=sqlite_autoindex_c_1, root=7, iDb=0
5     Rewind             0     33    0                    0   Rewind table a
6       Integer          0     4     0                    0   r[4]=0
7       Column           0     1     6                    0   r[6]=a.y
8       IsNull           6     28    0                    0   if (r[6]==NULL) goto 28
9       SeekGE           2     28    6                    0   key=[6..6]
10        IdxGT          2     28    6                    0   key=[6..6]
11        DeferredSeek   2     1     0                    0   
12        Integer        1     4     0                    0   r[4]=1
13        Integer        0     5     0                    0   r[5]=0
14        Column         1     1     7                    0   r[7]=b.y
-- if b.y is NULL, we skip the entire table loop between insns 16-23
-- except when we call NullRow and then Goto to re-enter that loop in order to
-- return NULL values for the table
15        IsNull         7     24    0                    0   if (r[7]==NULL) goto 24
16        SeekGE         3     24    7                    0   key=[7..7]
17          IdxGT        3     24    7                    0   key=[7..7]
18          Integer      1     5     0                    0   r[5]=1
19          Column       0     0     1                    0   r[1]=a.x
20          Column       1     0     2                    0   r[2]=b.x
21          Column       3     0     3                    0   r[3]=sqlite_autoindex_c_1.x
22          ResultRow    1     3     0                    0   output=r[1..3]
23        Next           3     17    0                    0   
24        IfPos          5     27    0                    0   r[5]>0 -> r[5]-=0, goto 27
25        NullRow        3     0     0                    0   Set cursor 3 to a (pseudo) NULL row
26        Goto           0     18    0                    0   
27      Next             2     10    0                    0   
28      IfPos            4     32    0                    0   r[4]>0 -> r[4]-=0, goto 32
29      NullRow          1     0     0                    0   Set cursor 1 to a (pseudo) NULL row
30      NullRow          2     0     0                    0   Set cursor 2 to a (pseudo) NULL row
31      Goto             0     12    0                    0   
32    Next               0     6     0                    0   
33    Halt               0     0     0                    0   
34    Transaction        0     0     3                    0   iDb=0 write=false
35    Goto               0     1     0                    0
```

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2967
2025-09-08 17:45:29 +03:00
Pekka Enberg
088cb449a5 Merge 'reduce cloning Arc<Page>' from Lâm Hoàng Phúc
only use `self.stack.top()` when we need to store `Arc<Page>` in struct
```sh
Execute `SELECT count() FROM users`/limbo_execute_select_count
                        time:   [5.3733 µs 5.3801 µs 5.3881 µs]
                        change: [-34.047% -33.949% -33.851%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
```

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2962
2025-09-08 17:02:35 +03:00
Pekka Enberg
d05ea4fd70 Merge 'cli: Fix dump compatibility in "PRAGMA foreign_keys"' from Pekka Enberg
SQLite emits a semicolon after "PRAGMA foreign_keys=OFF" so let's do th
same.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2961
2025-09-08 16:56:01 +03:00
Jussi Saurio
5820f691af fix: do not crash in Next if cursor stack has no pages 2025-09-08 16:54:35 +03:00
TcMits
3aa4650f06 make mr.clippy happy 2025-09-08 18:24:50 +07:00
Pekka Enberg
a2a5b852b8 Document query result ordering limitation 2025-09-08 14:19:48 +03:00
TcMits
a6ff568530 reduce cloning 'Arc<Page>' 2025-09-08 18:00:18 +07:00
Pekka Enberg
80a4358272 cli: Fix dump compatibility in "PRAGMA foreign_keys"
SQLite emits a semicolon after "PRAGMA foreign_keys=OFF" so let's do th
same.
2025-09-08 12:55:29 +03:00
Pekka Enberg
01879144b6 Merge 'Evaluate left join seek key condition again after null row' from Jussi Saurio
Closes #2949
This fixes a special case of the behavior described in #2501 - the
special case is that WHERE conditions that were selected as seek
predicates for the left join table were not properly evaluated when the
right-hand-side table returned no match.
The test in commit 12d72d115588a9e744bdb22382998ba1bf9031ab should
demonstrate this adequately - this should return no rows, but on `main`
it returns `1|NULL`.

Closes #2955
2025-09-08 12:01:17 +03:00
Pekka Enberg
93baef6ae4 Merge 'use mlugg/setup-zig instead of unmaintained action' from Kingsword
link: https://github.com/goto-bus-stop/setup-zig?tab=readme-ov-
file#setup-zig

Closes #2951
2025-09-08 12:00:57 +03:00
Jussi Saurio
c664639c09 Merge 'Add assertion: we read a page with the correct id' from Jussi Saurio
Part of debugging #2746 , but a good sanity check in any case.

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2802
2025-09-08 09:52:31 +03:00
Jussi Saurio
2c6e48903e Merge 'Prevent setting of encryption keys if already set' from Gaurav Sarma
Fixes https://github.com/tursodatabase/turso/issues/2883
<img width="867" height="128" alt="Screenshot 2025-09-05 at 10 44 18 PM"
src="https://github.com/user-attachments/assets/54a659ba-
cfe1-4622-939b-c7c31362ee5a" />

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2914
2025-09-08 09:49:55 +03:00
Jussi Saurio
b99a4f528d Merge 'support float without fractional part' from Lâm Hoàng Phúc
fix #2933

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2935
2025-09-08 09:49:20 +03:00
Jussi Saurio
03cb84ef30 Merge 'expr: use more efficient implementation for binary condition exprs' from Jussi Saurio
Closes #2946
currently we always evaluate the binary expression, then coerce it to
zero/null with the `ZeroOrNull` instruction, and then emit a separate
jump.
this is fine for non-conditional expressions where we are using the
value itself (e.g. in a SELECT result column), but in conditionals we
don't care about that at all and just want to either jump or not jump.
so: try to keep the spirit of code reuse, but still have distinct
implementations for conditionals and non-conditionals.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2947
2025-09-08 09:46:42 +03:00
Jussi Saurio
457f1980ca Merge 'Small fixes' from Nikita Sivukhin
Attempt to improve performance of `SELECT COUNT(*) FROM t WHERE 1 = 1`
(redundant filter added to force sqlite and tursodb to iterate over rows
instead of applying count optimization).
Before that PR turso with hot cache runs for ~440ms.
After that PR turso with hot cache runs for ~200ms.
This PR eliminates small things on a hot path and also introduce
separate flow for `get_next_record` which just increment cell idx if its
safe to do that - without much additional overhead.
Also, this PR eliminates RefCell/Cell from PageStack because there were
completely unnecessary.

Reviewed-by: Preston Thorpe <preston@turso.tech>
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2887
2025-09-08 09:45:35 +03:00
Jussi Saurio
68ee447d35 Minimal regression test for 2949 2025-09-08 09:35:11 +03:00
Jussi Saurio
f6ede79de7 Do not consume non-join WHERE terms in left joins 2025-09-08 09:35:11 +03:00
Nikita Sivukhin
87d49cd039 cargo fmt after rebase 2025-09-07 20:08:10 +04:00
Nikita Sivukhin
cd627c2368 remove unnecessary changes 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
47808e9da8 enable tracing subscriber in integration tests 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
5b9fe0cdf3 fix 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
0b6a6e7713 remove comma 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
9aed831f2f format 2025-09-07 19:56:05 +04:00
Nikita Sivukhin
db7c6b3370 try to speed up count(*) where 1 = 1 2025-09-07 19:55:42 +04:00
Nikita Sivukhin
c374cf0c93 remove Cell/RefCell from PageStack 2025-09-07 19:54:50 +04:00
Kingsword
1bef77ae42 use mlugg/setup-zig instead of unmaintained action 2025-09-07 17:50:30 +08:00
Preston Thorpe
b7c43cf293 Merge 'Add json_each table-valued function (1-arg only)' from Mikaël Francoeur
This adds the [`json_each`](https://sqlite.org/json1.html#the_json_each_
and_json_tree_table_valued_functions) TVF. Only the 1-arg version is
supported for now.
As suggested in the comments on this PR, I've also extended the virtual
table system to support internal TVF's, as opposed to extensions and
pragma TVF's.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2691
2025-09-06 19:46:20 -04:00
Jussi Saurio
ed3c73a194 kargo klippy 2025-09-06 17:51:15 +03:00
Jussi Saurio
430101ab47 expr: use more efficient implementation for binary condition exprs
currently we always evaluate the binary expression, then coerce it
to zero/null with the `ZeroOrNull` instruction, and then emit a separate
jump.

this is fine for non-conditional expressions where we are using the value
itself (e.g. in a SELECT result column), but in conditionals we don't care
about that at all and just want to jump.

so: try to keep the spirit of code reuse, but still have distinct implementations
for conditionals and non-conditionals.
2025-09-06 17:45:54 +03:00
Gaurav Sarma
b3242a18d9 Prevent setting of encryption keys if already set 2025-09-06 22:37:12 +08:00
Pekka Enberg
9c24b8d088 Merge 'Remove RefCell from Cursor' from Pedro Muniz
Closes #2944
2025-09-06 15:03:23 +03:00
pedrocarlo
e6344db5b1 remove Refcell from Cursor 2025-09-06 01:46:21 -03:00
Preston Thorpe
ec40ee6d3f Merge 'Page Cache: optimize and use sieve/Gclock hybird algorithm in place of LRU' from Preston Thorpe
This PR replaces the existing LRU cache implementation with the sieve
algo, providing better performance and simpler code.
Same intrusive linked-list without pointer chasing and no per-
page/insert heap allocations.
Benchmarks look really good so far, although some did show some
regression (I think it's more likely that they were flakey and its not
representative on either end, but I do believe this to be considerably
more performant)
Best numbers I have seen yet on Mobibench locally, with 1473 tx/s.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2885
2025-09-05 23:14:32 -04:00
PThorpe92
01d64977d7 Use more efficient circular list and rely on clock hand for pagecache 2025-09-05 22:40:27 -04:00
PThorpe92
644d0f270b Add evict slot method in page cache 2025-09-05 16:13:30 -04:00
PThorpe92
b89513f031 remove useless saturating sub 2025-09-05 16:13:30 -04:00
PThorpe92
39a47d67e6 Apply PR suggestions 2025-09-05 16:13:29 -04:00
PThorpe92
f45a7538fe Use true sieve/gclock algo instead of lru,dont link pages circilarly 2025-09-05 16:13:29 -04:00
PThorpe92
e418a902e5 Fix scoping issues now that refcells are gone to prevent extra destructors 2025-09-05 16:13:28 -04:00
PThorpe92
c85a61442f Remove type alias in page cache 2025-09-05 16:13:28 -04:00
PThorpe92
5ba273eea5 remove unused impl for refbit 2025-09-05 16:13:28 -04:00
PThorpe92
246b62d513 Remove unnecessary refcells, as PageCacheEntry has interior mutability 2025-09-05 16:13:27 -04:00