Commit Graph

4737 Commits

Author SHA1 Message Date
Pekka Enberg
c620a15a55 Merge 'core: Support ceiling modifier in datetime' from Ceferino Patino
Resolves #2677
- Implements the modifier for Floor on the datetime object.
- Implements the modifier for Ceiling on the datetime object.
- Includes additional testing changes in testing/scalar-functions-
datetime.test to test the sql functionality.
Consolidation PR of #2678 and #2679 since the functions ended up being
much more complicated than I initially thought and had to be done in a
single PR.

Closes #2757
2025-09-10 14:46:07 +03:00
Pekka Enberg
a3d23c2738 Merge 'Fix jsonb functions check valid json string binary' from
closes: #2820
```
turso> select user->>'age' as age from json_user;
┌─────┐
│ age │
├─────┤
│  30 │
└─────┘
```

Closes #2821
2025-09-10 14:44:54 +03:00
Pekka Enberg
0b91f8a715 Merge 'IO: handle errors properly in io_uring' from Preston Thorpe
Because `io_uring` may have many other I/O submission events queued
(that are relevant to the operation) when we experience an error,
marking our `Completion` objects as aborted is not sufficient, the
kernel will still execute queued I/O, which can mutate WAL or DB state
after we’ve declared failure and keep references (iovec arrays, buffers)
alive and stall reuse. We need to stop those in-flight SQEs at the
kernel and then drain the ring to a known-empty state before reusing any
resources.
The following methods were added to the `IO` trait:
`cancel`: which takes a slice of `Completion` objects and has a default
implementation that simply marks them as `aborted`.
`drain`: which has a default noop implementation, but the `io_uring`
backend implements this method to drain the ring.
CC @sivukhin

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2787
2025-09-10 14:24:43 +03:00
Pekka Enberg
22b352bc2f Merge 'core: Fix integer/float comparison' from Pavan Nambi
closes #2957

Closes #2958
2025-09-10 14:13:15 +03:00
Pekka Enberg
1ec6233fe1 Merge 'pager: fix incorrect freelist page count bookkeeping' from Jussi Saurio
When we use a freelist trunk page, the freelist count should decrease,
not increase
file under: lmao
Closes #2992

Closes #3000
2025-09-10 14:10:48 +03:00
Pekka Enberg
ab871b70b1 Merge 'core/vdbe: Don't rollback transaction when write upgrade fails' from Pekka Enberg
If upgrade from read to write transaction fails, don't roll back the
transaction. Instead restore the transaction into its original state,
which allows deferred transactions that have not read anything to
restart automatically.
Fixes #2984

Closes #2996
2025-09-10 14:09:29 +03:00
Jussi Saurio
5c8afc5caf pager: fix incorrect freelist page count bookkeeping 2025-09-10 14:02:17 +03:00
Jussi Saurio
11339fc941 Merge 'Fix clear_page_cache method and rollback' from Preston Thorpe
Previously we were iterating over every entry in the page cache,
clearing the dirty flag from each page.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2988
2025-09-10 11:11:37 +03:00
Jussi Saurio
ab9dbba17c Merge 'Fix read_entire_wal_dumb: incrementally build the frame cache' from Preston Thorpe
closes #2240
Incrementally build the frame cache by reading the WAL file in chunks
instead of reading the entire file into memory.
<img width="247" height="254" alt="image" src="https://github.com/user-
attachments/assets/803645ab-002a-4efd-ac47-b2f690e63fc7" />

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2986
2025-09-10 11:09:42 +03:00
Pekka Enberg
8161badbf4 core/vdbe: Don't rollback transaction when write upgrade fails
If upgrade from read to write transaction fails, don't roll back the
transaction. Instead restore the transaction into its original state,
which allows deferred transactions that have not read anything to
restart automatically.

Fixes #2984
2025-09-10 10:51:52 +03:00
Preston Thorpe
d32d19b47f Merge 'Implement 2-args json_each' from Mikaël Francoeur
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2953
2025-09-09 16:43:24 -04:00
PThorpe92
2f4f67efa8 Remove some unused attributes 2025-09-09 16:17:49 -04:00
PThorpe92
02bebf02a5 Remove read_entire_wal_dumb in favor of reading chunks 2025-09-09 16:06:27 -04:00
PThorpe92
cb12a1319d Fix page cache clear method to not re-initialize every slot 2025-09-09 15:55:59 -04:00
PThorpe92
8cc4e7f7a0 Fix rollback method to stop using highly inefficient cache::clear_dirty 2025-09-09 13:28:17 -04:00
PThorpe92
f7471a22c0 Fix clear_page_cache method and stop iterating over every entry 2025-09-09 13:25:33 -04:00
PThorpe92
37ec77eec2 Fix read_entire_wal_dumb to prefer streaming read if over 32mb wal file 2025-09-09 13:12:58 -04:00
Pekka Enberg
46c3bb21ad Merge 'Add OPFS support to JavaScript bindings' from Nikita Sivukhin
This PR restructure JS packages and also adds support for OPFS for
tursodatabase in browser.
The new structure looks like this:
1. `@tursodatabase/database-common` - contains abstract JS code for
bindings which depends only on `NativeDB` interface and not on the
explicit native bindings
2. `@tursodatabase/database` - contains native bindings for the database
and re-use `core` package
3. `@tursodatabase/database-browser` - contains bindings for browser
(WASM + OPFS)
As OPFS sync API (which is the most performant one in the web) works
only in the web worker - this PR also make few operations async in order
to run them as `napi-rs` AsyncTask. The following operations became
async in `promise.ts` for node and browser: `pragma` / `exec` / `close`.
Also, as few code pathes during initialization are non-async - they
complicates integration of sync constructor in the browser with OPFS.
So, right now - turso support only `connect` method for browser in non-
memory mode.

Closes #2927
2025-09-09 19:57:19 +03:00
Nikita Sivukhin
8b1b71d8b0 fix clippy 2025-09-09 12:21:54 +04:00
Nikita Sivukhin
4d80f8255f use MemoryIO for ephemeral tables for wasm target 2025-09-09 11:20:24 +04:00
Nikita Sivukhin
794440606a sligthly adjust attach to use same IO as main DB (if no custom VFS is specified) 2025-09-09 11:07:26 +04:00
Pekka Enberg
457aaeb1a7 Merge 'optimizer: convert outer join to inner join if possible' from Jussi Saurio
Convert `LEFT JOIN` to `INNER JOIN` when the result of `LEFT JOIN` can
never be different from the result of an `INNER JOIN`
This is useful because 1. it uses less instructions and 2. it allows for
join reordering due to inner join commutativity

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2972
2025-09-09 08:50:31 +03:00
Jussi Saurio
c930f28643 Handle case where null flag is set in op_column 2025-09-09 00:00:19 +03:00
Jussi Saurio
2d1d284279 optimizer: convert outer join to inner join if possible 2025-09-08 23:21:34 +03:00
PThorpe92
ccae3ab0f2 Change callsites to cancel any further IO when an error occurs and drain 2025-09-08 13:18:40 -04:00
PThorpe92
a750505762 Impl cancel and drain methods for io_uring 2025-09-08 13:18:38 -04:00
PThorpe92
eb0e069445 Add ShortWrite to CompletionError 2025-09-08 13:18:09 -04:00
PThorpe92
02df372811 Add cancel and drain methods to IO trait 2025-09-08 13:18:03 -04:00
Preston Thorpe
8d05336522 Merge 'Fix affinity handling in MakeRecord' from Pekka Enberg
Closes #2966
2025-09-08 12:14:33 -04:00
Pekka Enberg
71a812ce55 Merge 'Fix infinite loop when IO failure happens on allocating first page' from Preston Thorpe
closes #2919

Reviewed-by: Pedro Muniz (@pedrocarlo)

Closes #2968
2025-09-08 18:59:34 +03:00
PThorpe92
237b9fefd7 Fix infinite loop when IO failure happens on allocating first page 2025-09-08 11:49:33 -04:00
Pekka Enberg
0c6398c935 core/vdbe: Fix apply_affinity_char() text parsing
We need strict parsing in apply_affinity_char() to avoid transforming
non-numeric values (for example, "1a") into numeric values.
2025-09-08 18:49:13 +03:00
Pekka Enberg
f88f39082a core/vdbe: Fix MakeRecord affinity handling
The MakeRecord instruction now accepts an optional affinity_str
parameter that applies column-specific type conversions before creating
records. When provided, the affinity string is applied
character-by-character to each register using the existing
apply_affinity_char() function, matching SQLite's behavior.

Fixes #2040
Fixes #2041
2025-09-08 18:49:13 +03:00
Mikaël Francoeur
b480b526bc implement 2-args json_each 2025-09-08 11:34:17 -04:00
Pekka Enberg
081a7b563b Merge 'Fix crash in Next opcode if cursor stack has no pages' from Jussi Saurio
Closes #2924
Unsure if this fix is that great, but it does fix the issue described in
#2924 -- added minimal regression test to illustrate the behavior
This crash requires a pretty specific set of circumstances:
- 3-way join with two innermost being left joins
- nullable seek key on the innermost table:
    * middle table gets nulled out because no matches with the outermost
table
    * hence when we seek the innermost table using middle table values,
the seek key is null, so `Insn::IsNull` entirely skips the innermost
table
Perhaps a bytecode plan illustrates this better:
```sql
turso> explain select a.x, b.x, c.x from a left join b on a.y=b.x left join c on b.y=c.x;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     34    0                    0   Start at 34
1     OpenRead           0     2     0                    0   table=a, root=2, iDb=0
2     OpenRead           1     4     0                    0   table=b, root=4, iDb=0
3     OpenRead           2     5     0                    0   index=sqlite_autoindex_b_1, root=5, iDb=0
4     OpenRead           3     7     0                    0   index=sqlite_autoindex_c_1, root=7, iDb=0
5     Rewind             0     33    0                    0   Rewind table a
6       Integer          0     4     0                    0   r[4]=0
7       Column           0     1     6                    0   r[6]=a.y
8       IsNull           6     28    0                    0   if (r[6]==NULL) goto 28
9       SeekGE           2     28    6                    0   key=[6..6]
10        IdxGT          2     28    6                    0   key=[6..6]
11        DeferredSeek   2     1     0                    0   
12        Integer        1     4     0                    0   r[4]=1
13        Integer        0     5     0                    0   r[5]=0
14        Column         1     1     7                    0   r[7]=b.y
-- if b.y is NULL, we skip the entire table loop between insns 16-23
-- except when we call NullRow and then Goto to re-enter that loop in order to
-- return NULL values for the table
15        IsNull         7     24    0                    0   if (r[7]==NULL) goto 24
16        SeekGE         3     24    7                    0   key=[7..7]
17          IdxGT        3     24    7                    0   key=[7..7]
18          Integer      1     5     0                    0   r[5]=1
19          Column       0     0     1                    0   r[1]=a.x
20          Column       1     0     2                    0   r[2]=b.x
21          Column       3     0     3                    0   r[3]=sqlite_autoindex_c_1.x
22          ResultRow    1     3     0                    0   output=r[1..3]
23        Next           3     17    0                    0   
24        IfPos          5     27    0                    0   r[5]>0 -> r[5]-=0, goto 27
25        NullRow        3     0     0                    0   Set cursor 3 to a (pseudo) NULL row
26        Goto           0     18    0                    0   
27      Next             2     10    0                    0   
28      IfPos            4     32    0                    0   r[4]>0 -> r[4]-=0, goto 32
29      NullRow          1     0     0                    0   Set cursor 1 to a (pseudo) NULL row
30      NullRow          2     0     0                    0   Set cursor 2 to a (pseudo) NULL row
31      Goto             0     12    0                    0   
32    Next               0     6     0                    0   
33    Halt               0     0     0                    0   
34    Transaction        0     0     3                    0   iDb=0 write=false
35    Goto               0     1     0                    0
```

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2967
2025-09-08 17:45:29 +03:00
Pekka Enberg
088cb449a5 Merge 'reduce cloning Arc<Page>' from Lâm Hoàng Phúc
only use `self.stack.top()` when we need to store `Arc<Page>` in struct
```sh
Execute `SELECT count() FROM users`/limbo_execute_select_count
                        time:   [5.3733 µs 5.3801 µs 5.3881 µs]
                        change: [-34.047% -33.949% -33.851%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
```

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2962
2025-09-08 17:02:35 +03:00
Jussi Saurio
5820f691af fix: do not crash in Next if cursor stack has no pages 2025-09-08 16:54:35 +03:00
TcMits
3aa4650f06 make mr.clippy happy 2025-09-08 18:24:50 +07:00
TcMits
a6ff568530 reduce cloning 'Arc<Page>' 2025-09-08 18:00:18 +07:00
Pavan-Nambi
d757a330ee use sqlite_int_float_compare 2025-09-08 16:26:37 +05:30
Pekka Enberg
01879144b6 Merge 'Evaluate left join seek key condition again after null row' from Jussi Saurio
Closes #2949
This fixes a special case of the behavior described in #2501 - the
special case is that WHERE conditions that were selected as seek
predicates for the left join table were not properly evaluated when the
right-hand-side table returned no match.
The test in commit 12d72d115588a9e744bdb22382998ba1bf9031ab should
demonstrate this adequately - this should return no rows, but on `main`
it returns `1|NULL`.

Closes #2955
2025-09-08 12:01:17 +03:00
Jussi Saurio
c664639c09 Merge 'Add assertion: we read a page with the correct id' from Jussi Saurio
Part of debugging #2746 , but a good sanity check in any case.

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2802
2025-09-08 09:52:31 +03:00
Jussi Saurio
2c6e48903e Merge 'Prevent setting of encryption keys if already set' from Gaurav Sarma
Fixes https://github.com/tursodatabase/turso/issues/2883
<img width="867" height="128" alt="Screenshot 2025-09-05 at 10 44 18 PM"
src="https://github.com/user-attachments/assets/54a659ba-
cfe1-4622-939b-c7c31362ee5a" />

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2914
2025-09-08 09:49:55 +03:00
Jussi Saurio
03cb84ef30 Merge 'expr: use more efficient implementation for binary condition exprs' from Jussi Saurio
Closes #2946
currently we always evaluate the binary expression, then coerce it to
zero/null with the `ZeroOrNull` instruction, and then emit a separate
jump.
this is fine for non-conditional expressions where we are using the
value itself (e.g. in a SELECT result column), but in conditionals we
don't care about that at all and just want to either jump or not jump.
so: try to keep the spirit of code reuse, but still have distinct
implementations for conditionals and non-conditionals.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2947
2025-09-08 09:46:42 +03:00
Jussi Saurio
f6ede79de7 Do not consume non-join WHERE terms in left joins 2025-09-08 09:35:11 +03:00
Nikita Sivukhin
87d49cd039 cargo fmt after rebase 2025-09-07 20:08:10 +04:00
Nikita Sivukhin
cd627c2368 remove unnecessary changes 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
5b9fe0cdf3 fix 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
0b6a6e7713 remove comma 2025-09-07 19:56:06 +04:00
Nikita Sivukhin
9aed831f2f format 2025-09-07 19:56:05 +04:00