Commit Graph

8062 Commits

Author SHA1 Message Date
Pekka Enberg
e3cfd1b68e Merge 'sqlite3: Implement sqlite3_next_stmt()' from Pekka Enberg
Closes #2780
2025-08-25 17:51:56 +03:00
Pekka Enberg
1a4a53e6ea Merge 'core/io: Fix build on Android and iOS' from Pekka Enberg
Commit ebe6aa0d28 ("adjust cfg for unix
and linux IO") adjusted the I/O conditional compilation, but forgot that
Android and iOS are also part of Unix target family.
Fixes #2500

Closes #2776
2025-08-25 15:34:52 +03:00
Pekka Enberg
3f5878243f Merge 'Remove unnecessary argument from Pager::end_tx()' from Nikita Sivukhin
No need to pass `disable` flag to the `end_tx` method as it has that
info from connection itself

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2777
2025-08-25 15:34:41 +03:00
Preston Thorpe
040ceba2d6 Merge 'WAL txn: fix reads from DB file' from Nikita Sivukhin
- Transaction which was started with max_frame = 0 and
max_frame_read_lock_index = 0 can write to the WAL and in this case it
needs to read data back from WAL and not the DB file.
- Without cache spilling its hard to reproduce this issue for the turso-
db now, but I found this issue with sync-engine which do weird stuff
with the WAL which "simulates" cache spilling behaviour to some extent.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2735
2025-08-25 08:34:17 -04:00
Pekka Enberg
16547cb569 sqlite3: Implement sqlite3_next_stmt() 2025-08-25 14:44:55 +03:00
Pekka Enberg
7401d33784 Merge 'core/translate: Add support' from Pekka Enberg
Fixes #2263

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2774
2025-08-25 11:53:36 +03:00
Jussi Saurio
f0a32c2312 Merge 'refactor/btree: rewrite the find_free_cell() function' from Jussi Saurio
Based on #2509, and in a similar spirit.

Closes #2510
2025-08-25 11:41:37 +03:00
Nikita Sivukhin
f7ad55b680 remove unnecessary argument 2025-08-25 12:24:39 +04:00
Pekka Enberg
f452f54ce3 Merge 'sqlite3: Implement sqlite3_db_filename()' from Pekka Enberg
Closes #2771
2025-08-25 11:23:13 +03:00
Pekka Enberg
5fe5e1548b core/io: Fix build on Android and iOS
Commit ebe6aa0d28 ("adjust cfg for unix
and linux IO") adjusted the I/O conditional compilation, but forgot that
Android and iOS are also part of Unix target family.

Fixes #2500
2025-08-25 11:21:46 +03:00
Pekka Enberg
5d3780f25d core/translate: Add CREATE INDEX IF NOT EXISTS support
Fixes #2263
2025-08-25 11:12:41 +03:00
Nikita Sivukhin
7410403691 fix clippy 2025-08-25 11:38:59 +04:00
Nikita Sivukhin
c62b87d9b6 read from database file only if max_frame_read_lock_index is 0 and max_frame > min_frame
- transaction which was started with max_frame = 0 and max_frame_read_lock_index = 0
  can write to the WAL and in this case it needs to read data back from WAL
- without cache spilling its hard to reproduce this issue for the turso-db now,
  but I stumbled into this issue with sync-engine which do weird stuff with the WAL
  which "simulates" cache spilling behaviour to some extent
2025-08-25 11:36:58 +04:00
Pekka Enberg
262ead8240 Merge 'ANALYZE creates sqlite_stat1 if it doesn't exist' from Alex Miller
This is change 2/N of #656
This change replaces a bail_parse_error!() when sqlite_stat1 doesn't
exist with the appropriate codegen to create the table, and handle both
cases of the table existing or not existing.
SQLite's codegen looks like:
```
sqlite> create table stat_test(a,b,c);
sqlite> explain analyze stat_test;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     40    0                    0   Start at 40
1     ReadCookie     0     3     2                    0
2     If             3     5     0                    0
3     SetCookie      0     2     4                    0
4     SetCookie      0     5     1                    0
5     CreateBtree    0     2     1                    0   r[2]=root iDb=0 flags=1
6     OpenWrite      0     1     0     5              0   root=1 iDb=0
7     NewRowid       0     1     0                    0   r[1]=rowid
8     Blob           6     3     0                   0   r[3]= (len=6)
9     Insert         0     3     1                    8   intkey=r[1] data=r[3]
10    Close          0     0     0                    0
11    Close          0     0     0                    0
12    Null           0     4     5                    0   r[4..5]=NULL
13    Noop           4     0     4                    0
14    OpenWrite      3     1     0     5              0   root=1 iDb=0; sqlite_master
15    SeekRowid      3     17    1                    0   intkey=r[1]
16    Rowid          3     5     0                    0   r[5]= rowid of 3
17    IsNull         5     26    0                    0   if r[5]==NULL goto 26
18    String8        0     6     0     table          0   r[6]='table'
19    String8        0     7     0     sqlite_stat1   0   r[7]='sqlite_stat1'
20    String8        0     8     0     sqlite_stat1   0   r[8]='sqlite_stat1'
21    Copy           2     9     0                    0   r[9]=r[2]
22    String8        0     10    0     CREATE TABLE sqlite_stat1(tbl,idx,stat) 0   r[10]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
23    MakeRecord     6     5     4     BBBDB          0   r[4]=mkrec(r[6..10])
24    Delete         3     68    5                    0
25    Insert         3     4     5                    0   intkey=r[5] data=r[4]
26    SetCookie      0     1     2                    0
27    ParseSchema    0     0     0     tbl_name='sqlite_stat1' AND type!='trigger' 0
28    OpenWrite      0     2     0     3              16  root=2 iDb=0; sqlite_stat1
29    OpenRead       5     2     0     3              0   root=2 iDb=0; stat_test
30    String8        0     18    0     stat_test      0   r[18]='stat_test'; stat_test
31    Count          5     20    0                    0   r[20]=count()
32    IfNot          20    37    0                    0
33    Null           0     19    0                    0   r[19]=NULL
34    MakeRecord     18    3     16    BBB            0   r[16]=mkrec(r[18..20])
35    NewRowid       0     12    0                    0   r[12]=rowid
36    Insert         0     16    12                   8   intkey=r[12] data=r[16]
37    LoadAnalysis   0     0     0                    0
38    Expire         0     0     0                    0
39    Halt           0     0     0                    0
40    Transaction    0     1     1     0              1   usesStmtJournal=1
41    Goto           0     1     0                    0
```
And now Turso's looks like:
```
turso> create table stat_test(a,b,c);
turso> explain analyze stat_test;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     23    0                    0   Start at 23
1     Null               0     1     0                    0   r[1]=NULL
2     CreateBtree        0     2     1                    0   r[2]=root iDb=0 flags=1
3     OpenWrite          0     1     0                    0   root=1; iDb=0
4     NewRowid           0     3     0                    0   r[3]=rowid
5     String8            0     4     0     table          0   r[4]='table'
6     String8            0     5     0     sqlite_stat1   0   r[5]='sqlite_stat1'
7     String8            0     6     0     sqlite_stat1   0   r[6]='sqlite_stat1'
8     Copy               2     7     0                    0   r[7]=r[2]
9     String8            0     8     0     CREATE TABLE sqlite_stat1(tbl,idx,stat)  0   r[8]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
10    MakeRecord         4     5     9                    0   r[9]=mkrec(r[4..8])
11    Insert             0     9     3     sqlite_stat1   0   intkey=r[3] data=r[9]
12    ParseSchema        0     0     0     tbl_name = 'sqlite_stat1' AND type != 'trigger'  0   tbl_name = 'sqlite_stat1' AND type != 'trigger'
13    OpenWrite          1     2     0                    0   root=2; iDb=0
14    OpenRead           2     2     0                    0   =stat_test, root=2, iDb=0
15    String8            0     12    0     stat_test      0   r[12]='stat_test'
16    Count              2     14    0                    0
17    IfNot              14    22    0                    0   if !r[14] goto 22
18    Null               0     13    0                    0   r[13]=NULL
19    MakeRecord         12    3     11                   0   r[11]=mkrec(r[12..14])
20    NewRowid           1     10    0                    0   r[10]=rowid
21    Insert             1     11    10    sqlite_stat1   0   intkey=r[10] data=r[11]
22    Halt               0     0     0                    0
23    Goto               0     1     0                    0
```
The notable difference in size is following the same codegen difference
in CREATE TABLE, where sqlite's odd dance of adding a placeholder entry
which is immediately replaced is instead done in tursodb as just
inserting the correct row in the first place. Aside from lines 6-13 of
sqlite's vdbe being missing, there's still the lack of LoadAnalysis,
Expire, and Cookie management.

Closes #2765
2025-08-25 10:24:46 +03:00
Pekka Enberg
b162f89b73 sqlite3: Implement sqlite3_db_filename() 2025-08-25 10:20:50 +03:00
Pekka Enberg
4bca5edb9e Merge 'Don't clear transaction state in nested statement' from Jussi Saurio
Closes #2738
Closes #2739
Closes #2753
Closes #2755
Closes #2767
Closes #2768
Closes #2769
Closes #2770
## El problema
If a connection does e.g. CREATE TABLE, it will start a "child
statement" to reparse the schema. That statement does not start its own
transaction, and so should not try to end the existing one either.
We had a logic bug where these steps would happen:
- `CREATE TABLE` executed successfully
- pread fault happens inside `ParseSchema` child stmt
- `handle_program_error()` is called
- `pager.end_tx()` returns immediately because `is_nested_stmt` is true
and we correctly no-op it.
- however, crucially: `handle_program_error()` then sets tx state to
None
- parent statement now catches error from nested stmt and calls
`handle_program_error()`, which calls `pager.end_tx()` again, and since
txn state is None, when it calls `rollback()` we panic on the assertion
`"dirty pages should be empty for read txn"`
## La solucion
Do not do _any_ error processing in `handle_program_error()` inside a
nested stmt. This means that the parent write txn is still active when
it processes the error from the child and we avoid this panic.

Closes #2772
2025-08-25 10:12:09 +03:00
Jussi Saurio
dc6bcd4d41 refactor/btree: rewrite find_free_cell() 2025-08-25 10:08:39 +03:00
Jussi Saurio
1777440af7 Merge 'refactor/btree: rewrite the free_cell_range() function' from Jussi Saurio
i had a rough time reading this function earlier and trying to
understand it, so rewrote it in a way that, to me, is much more
readable.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2509
2025-08-25 10:08:11 +03:00
Jussi Saurio
4ea8cd0007 refactor/btree: rewrite the free_cell_range() function
i had a rough time reading this function earlier and trying to understand it,
so rewrote it in a way that, to me, is much more readable.
2025-08-25 09:41:44 +03:00
Jussi Saurio
aa07b81e48 Merge 'flake.nix: add uv dependency to nativeBuildInputs' from Ceferino Patino
`make test` fails when using the nix-shell environment due to `uv` not
being included in the list of dependencies with
`make: uv: No such file or directory
make: *** [Makefile:55: uv-sync-test] Error 127`
simple fix adding it to `nativeBuildInputs` of the shell. after that
runs as expected.

Reviewed-by: Levy A. (@levydsa)

Closes #2636
2025-08-25 09:18:15 +03:00
Jussi Saurio
f2598a2dea Merge 'Remove Result from signature' from Mikaël Francoeur
This PR removes `Result<()>` from `Jsonb::write_to_string()`, since it
wasn't required. The method now returns `()`.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2717
2025-08-25 09:17:37 +03:00
Jussi Saurio
54ff656c9d Do not clear txn state inside nested statement
If a connection does e.g. CREATE TABLE, it will start a "child statement"
to reparse the schema. That statement does not start its own transaction,
and so should not try to end the existing one either.

We had a logic bug where these steps would happen:

- `CREATE TABLE` executed successfully
- pread fault happens inside `ParseSchema` child stmt
- `handle_program_error()` is called
- `pager.end_tx()` returns immediately because `is_nested_stmt` is true
  and we correctly no-op it.
- however, crucially: `handle_program_error()` then sets tx state to None
- parent statement now catches error from nested stmt and calls
  `handle_program_error()`, which calls `pager.end_tx()` again, and since
  txn state is None, when it calls `rollback()` we panic on the assertion
 `"dirty pages should be empty for read txn"`

Solution:

Do not do _any_ error processing in `handle_program_error()` inside a nested
stmt. This means that the parent write txn is still active when it processes
the error from the child and we avoid this panic.
2025-08-25 08:49:22 +03:00
Pekka Enberg
2e11e1bb8c Merge 'Switch to F_FULLSYNC on Darwin' from
Closes #2743 .

Closes #2766
2025-08-25 08:09:29 +03:00
rajajisai
9068a29380 Use unsafe block 2025-08-24 18:56:05 -04:00
rajajisai
84d20ba60f Use F_FULLSYNC in darwin based operating systems 2025-08-24 18:45:46 -04:00
Alex Miller
370da9fa59 ANALYZE creates sqlite_stat1 if it doesn't exist
This change replaces a bail_parse_error!() when sqlite_stat1 doesn't
exist with the appropriate codegen to create the table, and handle both
cases of the table existing or not existing.

SQLite's codegen looks like:

sqlite> create table stat_test(a,b,c);
sqlite> explain analyze stat_test;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     40    0                    0   Start at 40
1     ReadCookie     0     3     2                    0
2     If             3     5     0                    0
3     SetCookie      0     2     4                    0
4     SetCookie      0     5     1                    0
5     CreateBtree    0     2     1                    0   r[2]=root iDb=0 flags=1
6     OpenWrite      0     1     0     5              0   root=1 iDb=0
7     NewRowid       0     1     0                    0   r[1]=rowid
8     Blob           6     3     0                   0   r[3]= (len=6)
9     Insert         0     3     1                    8   intkey=r[1] data=r[3]
10    Close          0     0     0                    0
11    Close          0     0     0                    0
12    Null           0     4     5                    0   r[4..5]=NULL
13    Noop           4     0     4                    0
14    OpenWrite      3     1     0     5              0   root=1 iDb=0; sqlite_master
15    SeekRowid      3     17    1                    0   intkey=r[1]
16    Rowid          3     5     0                    0   r[5]= rowid of 3
17    IsNull         5     26    0                    0   if r[5]==NULL goto 26
18    String8        0     6     0     table          0   r[6]='table'
19    String8        0     7     0     sqlite_stat1   0   r[7]='sqlite_stat1'
20    String8        0     8     0     sqlite_stat1   0   r[8]='sqlite_stat1'
21    Copy           2     9     0                    0   r[9]=r[2]
22    String8        0     10    0     CREATE TABLE sqlite_stat1(tbl,idx,stat) 0   r[10]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
23    MakeRecord     6     5     4     BBBDB          0   r[4]=mkrec(r[6..10])
24    Delete         3     68    5                    0
25    Insert         3     4     5                    0   intkey=r[5] data=r[4]
26    SetCookie      0     1     2                    0
27    ParseSchema    0     0     0     tbl_name='sqlite_stat1' AND type!='trigger' 0
28    OpenWrite      0     2     0     3              16  root=2 iDb=0; sqlite_stat1
29    OpenRead       5     2     0     3              0   root=2 iDb=0; stat_test
30    String8        0     18    0     stat_test      0   r[18]='stat_test'; stat_test
31    Count          5     20    0                    0   r[20]=count()
32    IfNot          20    37    0                    0
33    Null           0     19    0                    0   r[19]=NULL
34    MakeRecord     18    3     16    BBB            0   r[16]=mkrec(r[18..20])
35    NewRowid       0     12    0                    0   r[12]=rowid
36    Insert         0     16    12                   8   intkey=r[12] data=r[16]
37    LoadAnalysis   0     0     0                    0
38    Expire         0     0     0                    0
39    Halt           0     0     0                    0
40    Transaction    0     1     1     0              1   usesStmtJournal=1
41    Goto           0     1     0                    0

And now Turso's looks like:

turso> create table stat_test(a,b,c);
turso> explain analyze stat_test;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     23    0                    0   Start at 23
1     Null               0     1     0                    0   r[1]=NULL
2     CreateBtree        0     2     1                    0   r[2]=root iDb=0 flags=1
3     OpenWrite          0     1     0                    0   root=1; iDb=0
4     NewRowid           0     3     0                    0   r[3]=rowid
5     String8            0     4     0     table          0   r[4]='table'
6     String8            0     5     0     sqlite_stat1   0   r[5]='sqlite_stat1'
7     String8            0     6     0     sqlite_stat1   0   r[6]='sqlite_stat1'
8     Copy               2     7     0                    0   r[7]=r[2]
9     String8            0     8     0     CREATE TABLE sqlite_stat1(tbl,idx,stat)  0   r[8]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
10    MakeRecord         4     5     9                    0   r[9]=mkrec(r[4..8])
11    Insert             0     9     3     sqlite_stat1   0   intkey=r[3] data=r[9]
12    ParseSchema        0     0     0     tbl_name = 'sqlite_stat1' AND type != 'trigger'  0   tbl_name = 'sqlite_stat1' AND type != 'trigger'
13    OpenWrite          1     2     0                    0   root=2; iDb=0
14    OpenRead           2     2     0                    0   =stat_test, root=2, iDb=0
15    String8            0     12    0     stat_test      0   r[12]='stat_test'
16    Count              2     14    0                    0
17    IfNot              14    22    0                    0   if !r[14] goto 22
18    Null               0     13    0                    0   r[13]=NULL
19    MakeRecord         12    3     11                   0   r[11]=mkrec(r[12..14])
20    NewRowid           1     10    0                    0   r[10]=rowid
21    Insert             1     11    10    sqlite_stat1   0   intkey=r[10] data=r[11]
22    Halt               0     0     0                    0
23    Goto               0     1     0                    0

The notable difference in size is following the same codegen difference
in CREATE TABLE, where sqlite's odd dance of adding a placeholder entry
which is immediately replaced is instead done in tursodb as just
inserting the correct row in the first place. Aside from lines 6-13 of
sqlite's vdbe being missing, there's still the lack of LoadAnalysis,
Expire, and Cookie management.
2025-08-24 13:35:39 -07:00
Pekka Enberg
de16c770c3 Merge 'Remove duplicated attribute in ' from bit-aloo
Removed the redundant `#![allow(clippy::arc_with_non_send_sync)]` from
wal.rs, since it’s already defined in `mod.rs`.

Closes #2764
2025-08-24 20:54:26 +03:00
Pekka Enberg
7291b59418 Merge 'sqlite3: Implement sqlite3_bind_parameter_index()' from Pekka Enberg
Closes #2763
2025-08-24 20:54:18 +03:00
bit-aloo
37cebb0669 fix(clippy): remove duplicate arc_with_non_send_sync attribute in wal.rs 2025-08-24 22:59:47 +05:30
Pekka Enberg
c428ff06b2 sqlite3: Implement sqlite3_bind_parameter_index() 2025-08-24 20:10:31 +03:00
Pekka Enberg
4fc7b94a6b Merge 'sqlite3: Implement sqlite3_clear_bindings()' from Pekka Enberg
Closes #2759
2025-08-24 19:57:14 +03:00
Pekka Enberg
9d2f26bb04 sqlite3: Implement sqlite3_clear_bindings() 2025-08-24 19:33:18 +03:00
Pekka Enberg
5a8f281555 Merge 'sqlite3: Implement sqlite3_get_autocommit()' from Pekka Enberg
Closes #2758
2025-08-24 16:53:04 +03:00
Pekka Enberg
a8ab049a36 Merge 'Add support for AEGIS encryption algorithm' from Avinash Sajjanshetty
Depends on #2722
This builds upon the #2722 PR which let me configure the encryption
algorithm. So this adds AEGIS and uses it as default.
Note that choice of cipher at higher APIs is still not possible. I have
a follow up PR which updates the PRAGMAs
AEGIS is way too damn fast, here are some numbers:
```
* MACs:

aegis128x4-mac      : 223.91 Gb/s
aegis128x2-mac      : 270.87 Gb/s
aegis128l-mac       : 229.35 Gb/s
sthash              : 83.60 Gb/s
hmac-sha256 (boring): 27.46 Gb/s
blake3              : 21.41 Gb/s

* Encryption:

aegis128x4          : 104.19 Gb/s
aegis128x2          : 182.46 Gb/s
aegis128l           : 181.62 Gb/s
aegis256x2          : 133.45 Gb/s
aegis256x4          : 125.23 Gb/s
aegis256            : 102.12 Gb/s
aes128-gcm (aes-gcm): 2.16 Gb/s
aes128-gcm (boring) : 63.25 Gb/s
aes256-gcm (aes-gcm): 1.70 Gb/s
aes256-gcm (boring) : 59.14 Gb/s
chacha20-poly1305   : 2.39 Gb/s
ascon128a           : 5.84 Gb/s
```

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2742
2025-08-24 14:16:10 +03:00
Pekka Enberg
f5dfbfffc2 Merge branch 'main' into aegis 2025-08-24 14:16:01 +03:00
Pekka Enberg
ea2192c332 sqlite3: Implement sqlite3_get_autocommit() 2025-08-24 14:13:36 +03:00
Pekka Enberg
2c6fa76437 cargo fmt 2025-08-24 14:13:20 +03:00
Avinash Sajjanshetty
011f878158 make clippy bro happy 2025-08-24 16:21:06 +05:30
Avinash Sajjanshetty
77a4e96022 run encryption tests in CI 2025-08-24 16:15:13 +05:30
Avinash Sajjanshetty
a4b9c33b81 Use the new API to init cipher 2025-08-24 16:15:13 +05:30
Avinash Sajjanshetty
53f9c0dc7a Add support for lord AEGIS, the fastest and the greatest 2025-08-24 16:15:11 +05:30
Pekka Enberg
73414663f8 Merge 'bindings/java: Implement batch operations for JDBC4Statement' from Kim Seon Woo
Implement the following methods for JDBC4Statement:
- addBatch
- clearBatch
- executeBatch

Closes #2754
2025-08-24 08:54:44 +03:00
Pekka Enberg
147740c95e Merge 'reduce cloning Token in parser' from Lâm Hoàng Phúc
small changes make parser 3-6% faster

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2745
2025-08-24 08:53:35 +03:00
Pekka Enberg
22c9cb6618 s/PerConnEncryptionContext/EncryptionContext/ 2025-08-24 08:17:20 +03:00
Pekka Enberg
1b89273f10 Merge 'refactor encryption module and make it configurable' from Avinash Sajjanshetty
Previously, the encryption module had hardcoded a lot of things. This
refactor makes it slightly nice and makes it configurable.
Right now cipher algorithm is assumed and hardcoded, I will make that
configurable in the upcoming PR

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2722
2025-08-24 08:16:28 +03:00
김선우
7057c97cfe Remove .rustc_info.json 2025-08-24 10:25:14 +09:00
김선우
fa8896d9ee Nit 2025-08-24 10:20:39 +09:00
김선우
9f6eb8bc92 Update verification of batch compatible statements using regex 2025-08-24 10:13:04 +09:00
김선우
bf1473dc08 Override JDBC4PreparedStatement to throw exception when calling addBatch method 2025-08-24 09:35:29 +09:00
김선우
346525e5f0 Update test 2025-08-24 09:25:59 +09:00