Commit Graph

4722 Commits

Author SHA1 Message Date
Jussi Saurio
11339fc941 Merge 'Fix clear_page_cache method and rollback' from Preston Thorpe
Previously we were iterating over every entry in the page cache,
clearing the dirty flag from each page.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2988
2025-09-10 11:11:37 +03:00
Jussi Saurio
ab9dbba17c Merge 'Fix read_entire_wal_dumb: incrementally build the frame cache' from Preston Thorpe
closes #2240
Incrementally build the frame cache by reading the WAL file in chunks
instead of reading the entire file into memory.
<img width="247" height="254" alt="image" src="https://github.com/user-
attachments/assets/803645ab-002a-4efd-ac47-b2f690e63fc7" />

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2986
2025-09-10 11:09:42 +03:00
Preston Thorpe
d32d19b47f Merge 'Implement 2-args json_each' from Mikaël Francoeur
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2953
2025-09-09 16:43:24 -04:00
PThorpe92
2f4f67efa8 Remove some unused attributes 2025-09-09 16:17:49 -04:00
PThorpe92
02bebf02a5 Remove read_entire_wal_dumb in favor of reading chunks 2025-09-09 16:06:27 -04:00
PThorpe92
cb12a1319d Fix page cache clear method to not re-initialize every slot 2025-09-09 15:55:59 -04:00
PThorpe92
8cc4e7f7a0 Fix rollback method to stop using highly inefficient cache::clear_dirty 2025-09-09 13:28:17 -04:00
PThorpe92
f7471a22c0 Fix clear_page_cache method and stop iterating over every entry 2025-09-09 13:25:33 -04:00
PThorpe92
37ec77eec2 Fix read_entire_wal_dumb to prefer streaming read if over 32mb wal file 2025-09-09 13:12:58 -04:00
Pekka Enberg
46c3bb21ad Merge 'Add OPFS support to JavaScript bindings' from Nikita Sivukhin
This PR restructure JS packages and also adds support for OPFS for
tursodatabase in browser.
The new structure looks like this:
1. `@tursodatabase/database-common` - contains abstract JS code for
bindings which depends only on `NativeDB` interface and not on the
explicit native bindings
2. `@tursodatabase/database` - contains native bindings for the database
and re-use `core` package
3. `@tursodatabase/database-browser` - contains bindings for browser
(WASM + OPFS)
As OPFS sync API (which is the most performant one in the web) works
only in the web worker - this PR also make few operations async in order
to run them as `napi-rs` AsyncTask. The following operations became
async in `promise.ts` for node and browser: `pragma` / `exec` / `close`.
Also, as few code pathes during initialization are non-async - they
complicates integration of sync constructor in the browser with OPFS.
So, right now - turso support only `connect` method for browser in non-
memory mode.

Closes #2927
2025-09-09 19:57:19 +03:00
Nikita Sivukhin
8b1b71d8b0 fix clippy 2025-09-09 12:21:54 +04:00
Nikita Sivukhin
4d80f8255f use MemoryIO for ephemeral tables for wasm target 2025-09-09 11:20:24 +04:00
Nikita Sivukhin
794440606a sligthly adjust attach to use same IO as main DB (if no custom VFS is specified) 2025-09-09 11:07:26 +04:00
Pekka Enberg
457aaeb1a7 Merge 'optimizer: convert outer join to inner join if possible' from Jussi Saurio
Convert `LEFT JOIN` to `INNER JOIN` when the result of `LEFT JOIN` can
never be different from the result of an `INNER JOIN`
This is useful because 1. it uses less instructions and 2. it allows for
join reordering due to inner join commutativity

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2972
2025-09-09 08:50:31 +03:00
Jussi Saurio
c930f28643 Handle case where null flag is set in op_column 2025-09-09 00:00:19 +03:00
Jussi Saurio
2d1d284279 optimizer: convert outer join to inner join if possible 2025-09-08 23:21:34 +03:00
Preston Thorpe
8d05336522 Merge 'Fix affinity handling in MakeRecord' from Pekka Enberg
Closes #2966
2025-09-08 12:14:33 -04:00
Pekka Enberg
71a812ce55 Merge 'Fix infinite loop when IO failure happens on allocating first page' from Preston Thorpe
closes #2919

Reviewed-by: Pedro Muniz (@pedrocarlo)

Closes #2968
2025-09-08 18:59:34 +03:00
PThorpe92
237b9fefd7 Fix infinite loop when IO failure happens on allocating first page 2025-09-08 11:49:33 -04:00
Pekka Enberg
0c6398c935 core/vdbe: Fix apply_affinity_char() text parsing
We need strict parsing in apply_affinity_char() to avoid transforming
non-numeric values (for example, "1a") into numeric values.
2025-09-08 18:49:13 +03:00
Pekka Enberg
f88f39082a core/vdbe: Fix MakeRecord affinity handling
The MakeRecord instruction now accepts an optional affinity_str
parameter that applies column-specific type conversions before creating
records. When provided, the affinity string is applied
character-by-character to each register using the existing
apply_affinity_char() function, matching SQLite's behavior.

Fixes #2040
Fixes #2041
2025-09-08 18:49:13 +03:00
Mikaël Francoeur
b480b526bc implement 2-args json_each 2025-09-08 11:34:17 -04:00
Pekka Enberg
081a7b563b Merge 'Fix crash in Next opcode if cursor stack has no pages' from Jussi Saurio
Closes #2924
Unsure if this fix is that great, but it does fix the issue described in
#2924 -- added minimal regression test to illustrate the behavior
This crash requires a pretty specific set of circumstances:
- 3-way join with two innermost being left joins
- nullable seek key on the innermost table:
    * middle table gets nulled out because no matches with the outermost
table
    * hence when we seek the innermost table using middle table values,
the seek key is null, so `Insn::IsNull` entirely skips the innermost
table
Perhaps a bytecode plan illustrates this better:
```sql
turso> explain select a.x, b.x, c.x from a left join b on a.y=b.x left join c on b.y=c.x;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     34    0                    0   Start at 34
1     OpenRead           0     2     0                    0   table=a, root=2, iDb=0
2     OpenRead           1     4     0                    0   table=b, root=4, iDb=0
3     OpenRead           2     5     0                    0   index=sqlite_autoindex_b_1, root=5, iDb=0
4     OpenRead           3     7     0                    0   index=sqlite_autoindex_c_1, root=7, iDb=0
5     Rewind             0     33    0                    0   Rewind table a
6       Integer          0     4     0                    0   r[4]=0
7       Column           0     1     6                    0   r[6]=a.y
8       IsNull           6     28    0                    0   if (r[6]==NULL) goto 28
9       SeekGE           2     28    6                    0   key=[6..6]
10        IdxGT          2     28    6                    0   key=[6..6]
11        DeferredSeek   2     1     0                    0   
12        Integer        1     4     0                    0   r[4]=1
13        Integer        0     5     0                    0   r[5]=0
14        Column         1     1     7                    0   r[7]=b.y
-- if b.y is NULL, we skip the entire table loop between insns 16-23
-- except when we call NullRow and then Goto to re-enter that loop in order to
-- return NULL values for the table
15        IsNull         7     24    0                    0   if (r[7]==NULL) goto 24
16        SeekGE         3     24    7                    0   key=[7..7]
17          IdxGT        3     24    7                    0   key=[7..7]
18          Integer      1     5     0                    0   r[5]=1
19          Column       0     0     1                    0   r[1]=a.x
20          Column       1     0     2                    0   r[2]=b.x
21          Column       3     0     3                    0   r[3]=sqlite_autoindex_c_1.x
22          ResultRow    1     3     0                    0   output=r[1..3]
23        Next           3     17    0                    0   
24        IfPos          5     27    0                    0   r[5]>0 -> r[5]-=0, goto 27
25        NullRow        3     0     0                    0   Set cursor 3 to a (pseudo) NULL row
26        Goto           0     18    0                    0   
27      Next             2     10    0                    0   
28      IfPos            4     32    0                    0   r[4]>0 -> r[4]-=0, goto 32
29      NullRow          1     0     0                    0   Set cursor 1 to a (pseudo) NULL row
30      NullRow          2     0     0                    0   Set cursor 2 to a (pseudo) NULL row
31      Goto             0     12    0                    0   
32    Next               0     6     0                    0   
33    Halt               0     0     0                    0   
34    Transaction        0     0     3                    0   iDb=0 write=false
35    Goto               0     1     0                    0
```

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2967
2025-09-08 17:45:29 +03:00
Pekka Enberg
088cb449a5 Merge 'reduce cloning Arc<Page>' from Lâm Hoàng Phúc
only use `self.stack.top()` when we need to store `Arc<Page>` in struct
```sh
Execute `SELECT count() FROM users`/limbo_execute_select_count
                        time:   [5.3733 µs 5.3801 µs 5.3881 µs]
                        change: [-34.047% -33.949% -33.851%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
```

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2962
2025-09-08 17:02:35 +03:00
Jussi Saurio
5820f691af fix: do not crash in Next if cursor stack has no pages 2025-09-08 16:54:35 +03:00
TcMits
3aa4650f06 make mr.clippy happy 2025-09-08 18:24:50 +07:00
TcMits
a6ff568530 reduce cloning 'Arc<Page>' 2025-09-08 18:00:18 +07:00
Pekka Enberg
01879144b6 Merge 'Evaluate left join seek key condition again after null row' from Jussi Saurio
Closes #2949
This fixes a special case of the behavior described in #2501 - the
special case is that WHERE conditions that were selected as seek
predicates for the left join table were not properly evaluated when the
right-hand-side table returned no match.
The test in commit 12d72d115588a9e744bdb22382998ba1bf9031ab should
demonstrate this adequately - this should return no rows, but on `main`
it returns `1|NULL`.

Closes #2955
2025-09-08 12:01:17 +03:00
Jussi Saurio
c664639c09 Merge 'Add assertion: we read a page with the correct id' from Jussi Saurio
Part of debugging #2746 , but a good sanity check in any case.

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2802
2025-09-08 09:52:31 +03:00
Jussi Saurio
2c6e48903e Merge 'Prevent setting of encryption keys if already set' from Gaurav Sarma
Fixes https://github.com/tursodatabase/turso/issues/2883
<img width="867" height="128" alt="Screenshot 2025-09-05 at 10 44 18 PM"
src="https://github.com/user-attachments/assets/54a659ba-
cfe1-4622-939b-c7c31362ee5a" />

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2914
2025-09-08 09:49:55 +03:00
Jussi Saurio
03cb84ef30 Merge 'expr: use more efficient implementation for binary condition exprs' from Jussi Saurio
Closes #2946
currently we always evaluate the binary expression, then coerce it to
zero/null with the `ZeroOrNull` instruction, and then emit a separate
jump.
this is fine for non-conditional expressions where we are using the
value itself (e.g. in a SELECT result column), but in conditionals we
don't care about that at all and just want to either jump or not jump.
so: try to keep the spirit of code reuse, but still have distinct
implementations for conditionals and non-conditionals.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2947
2025-09-08 09:46:42 +03:00
Jussi Saurio
f6ede79de7 Do not consume non-join WHERE terms in left joins 2025-09-08 09:35:11 +03:00
Nikita Sivukhin
87d49cd039 cargo fmt after rebase 2025-09-07 20:08:10 +04:00
Nikita Sivukhin
cd627c2368 remove unnecessary changes 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
5b9fe0cdf3 fix 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
0b6a6e7713 remove comma 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
9aed831f2f format 2025-09-07 19:56:05 +04:00
Nikita Sivukhin
db7c6b3370 try to speed up count(*) where 1 = 1 2025-09-07 19:55:42 +04:00
Nikita Sivukhin
c374cf0c93 remove Cell/RefCell from PageStack 2025-09-07 19:54:50 +04:00
Preston Thorpe
b7c43cf293 Merge 'Add json_each table-valued function (1-arg only)' from Mikaël Francoeur
This adds the [`json_each`](https://sqlite.org/json1.html#the_json_each_
and_json_tree_table_valued_functions) TVF. Only the 1-arg version is
supported for now.
As suggested in the comments on this PR, I've also extended the virtual
table system to support internal TVF's, as opposed to extensions and
pragma TVF's.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2691
2025-09-06 19:46:20 -04:00
Jussi Saurio
ed3c73a194 kargo klippy 2025-09-06 17:51:15 +03:00
Jussi Saurio
430101ab47 expr: use more efficient implementation for binary condition exprs
currently we always evaluate the binary expression, then coerce it
to zero/null with the `ZeroOrNull` instruction, and then emit a separate
jump.

this is fine for non-conditional expressions where we are using the value
itself (e.g. in a SELECT result column), but in conditionals we don't care
about that at all and just want to jump.

so: try to keep the spirit of code reuse, but still have distinct implementations
for conditionals and non-conditionals.
2025-09-06 17:45:54 +03:00
Gaurav Sarma
b3242a18d9 Prevent setting of encryption keys if already set 2025-09-06 22:37:12 +08:00
Pekka Enberg
9c24b8d088 Merge 'Remove RefCell from Cursor' from Pedro Muniz
Closes #2944
2025-09-06 15:03:23 +03:00
pedrocarlo
e6344db5b1 remove Refcell from Cursor 2025-09-06 01:46:21 -03:00
PThorpe92
01d64977d7 Use more efficient circular list and rely on clock hand for pagecache 2025-09-05 22:40:27 -04:00
PThorpe92
644d0f270b Add evict slot method in page cache 2025-09-05 16:13:30 -04:00
PThorpe92
b89513f031 remove useless saturating sub 2025-09-05 16:13:30 -04:00
PThorpe92
39a47d67e6 Apply PR suggestions 2025-09-05 16:13:29 -04:00
PThorpe92
f45a7538fe Use true sieve/gclock algo instead of lru,dont link pages circilarly 2025-09-05 16:13:29 -04:00