rajajisai
84d20ba60f
Use F_FULLSYNC in darwin based operating systems
2025-08-24 18:45:46 -04:00
Avinash Sajjanshetty
48ce2a4a3e
Set encryption ctx when cipher and key are set
2025-08-25 02:28:57 +05:30
Avinash Sajjanshetty
328c5edf4d
Add PRAGMA cipher to allow setting cipher algo
2025-08-25 02:17:53 +05:30
Alex Miller
370da9fa59
ANALYZE creates sqlite_stat1 if it doesn't exist
...
This change replaces a bail_parse_error!() when sqlite_stat1 doesn't
exist with the appropriate codegen to create the table, and handle both
cases of the table existing or not existing.
SQLite's codegen looks like:
sqlite> create table stat_test(a,b,c);
sqlite> explain analyze stat_test;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 40 0 0 Start at 40
1 ReadCookie 0 3 2 0
2 If 3 5 0 0
3 SetCookie 0 2 4 0
4 SetCookie 0 5 1 0
5 CreateBtree 0 2 1 0 r[2]=root iDb=0 flags=1
6 OpenWrite 0 1 0 5 0 root=1 iDb=0
7 NewRowid 0 1 0 0 r[1]=rowid
8 Blob 6 3 0 0 r[3]= (len=6)
9 Insert 0 3 1 8 intkey=r[1] data=r[3]
10 Close 0 0 0 0
11 Close 0 0 0 0
12 Null 0 4 5 0 r[4..5]=NULL
13 Noop 4 0 4 0
14 OpenWrite 3 1 0 5 0 root=1 iDb=0; sqlite_master
15 SeekRowid 3 17 1 0 intkey=r[1]
16 Rowid 3 5 0 0 r[5]= rowid of 3
17 IsNull 5 26 0 0 if r[5]==NULL goto 26
18 String8 0 6 0 table 0 r[6]='table'
19 String8 0 7 0 sqlite_stat1 0 r[7]='sqlite_stat1'
20 String8 0 8 0 sqlite_stat1 0 r[8]='sqlite_stat1'
21 Copy 2 9 0 0 r[9]=r[2]
22 String8 0 10 0 CREATE TABLE sqlite_stat1(tbl,idx,stat) 0 r[10]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
23 MakeRecord 6 5 4 BBBDB 0 r[4]=mkrec(r[6..10])
24 Delete 3 68 5 0
25 Insert 3 4 5 0 intkey=r[5] data=r[4]
26 SetCookie 0 1 2 0
27 ParseSchema 0 0 0 tbl_name='sqlite_stat1' AND type!='trigger' 0
28 OpenWrite 0 2 0 3 16 root=2 iDb=0; sqlite_stat1
29 OpenRead 5 2 0 3 0 root=2 iDb=0; stat_test
30 String8 0 18 0 stat_test 0 r[18]='stat_test'; stat_test
31 Count 5 20 0 0 r[20]=count()
32 IfNot 20 37 0 0
33 Null 0 19 0 0 r[19]=NULL
34 MakeRecord 18 3 16 BBB 0 r[16]=mkrec(r[18..20])
35 NewRowid 0 12 0 0 r[12]=rowid
36 Insert 0 16 12 8 intkey=r[12] data=r[16]
37 LoadAnalysis 0 0 0 0
38 Expire 0 0 0 0
39 Halt 0 0 0 0
40 Transaction 0 1 1 0 1 usesStmtJournal=1
41 Goto 0 1 0 0
And now Turso's looks like:
turso> create table stat_test(a,b,c);
turso> explain analyze stat_test;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 23 0 0 Start at 23
1 Null 0 1 0 0 r[1]=NULL
2 CreateBtree 0 2 1 0 r[2]=root iDb=0 flags=1
3 OpenWrite 0 1 0 0 root=1; iDb=0
4 NewRowid 0 3 0 0 r[3]=rowid
5 String8 0 4 0 table 0 r[4]='table'
6 String8 0 5 0 sqlite_stat1 0 r[5]='sqlite_stat1'
7 String8 0 6 0 sqlite_stat1 0 r[6]='sqlite_stat1'
8 Copy 2 7 0 0 r[7]=r[2]
9 String8 0 8 0 CREATE TABLE sqlite_stat1(tbl,idx,stat) 0 r[8]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
10 MakeRecord 4 5 9 0 r[9]=mkrec(r[4..8])
11 Insert 0 9 3 sqlite_stat1 0 intkey=r[3] data=r[9]
12 ParseSchema 0 0 0 tbl_name = 'sqlite_stat1' AND type != 'trigger' 0 tbl_name = 'sqlite_stat1' AND type != 'trigger'
13 OpenWrite 1 2 0 0 root=2; iDb=0
14 OpenRead 2 2 0 0 =stat_test, root=2, iDb=0
15 String8 0 12 0 stat_test 0 r[12]='stat_test'
16 Count 2 14 0 0
17 IfNot 14 22 0 0 if !r[14] goto 22
18 Null 0 13 0 0 r[13]=NULL
19 MakeRecord 12 3 11 0 r[11]=mkrec(r[12..14])
20 NewRowid 1 10 0 0 r[10]=rowid
21 Insert 1 11 10 sqlite_stat1 0 intkey=r[10] data=r[11]
22 Halt 0 0 0 0
23 Goto 0 1 0 0
The notable difference in size is following the same codegen difference
in CREATE TABLE, where sqlite's odd dance of adding a placeholder entry
which is immediately replaced is instead done in tursodb as just
inserting the correct row in the first place. Aside from lines 6-13 of
sqlite's vdbe being missing, there's still the lack of LoadAnalysis,
Expire, and Cookie management.
2025-08-24 13:35:39 -07:00
Avinash Sajjanshetty
279bcd0869
Remove unsecure EncryptionKey::from_string
2025-08-25 01:46:44 +05:30
Avinash Sajjanshetty
0308374d3a
Use proper hexadecimal key for encryption
...
Added `from_hex_string` which gets us `EncryptionKey` from a
hex string. Now we can use securely generated keys, like from openssl
$ openssl rand -hex 32
2025-08-25 01:36:05 +05:30
Avinash Sajjanshetty
543025f57a
rename encryption PRAGMA key to hexkey
2025-08-25 01:32:41 +05:30
Pekka Enberg
de16c770c3
Merge 'Remove duplicated attribute in ' from bit-aloo
...
Removed the redundant `#![allow(clippy::arc_with_non_send_sync)]` from
wal.rs, since it’s already defined in `mod.rs`.
Closes #2764
2025-08-24 20:54:26 +03:00
Pekka Enberg
7291b59418
Merge 'sqlite3: Implement sqlite3_bind_parameter_index()' from Pekka Enberg
...
Closes #2763
2025-08-24 20:54:18 +03:00
bit-aloo
37cebb0669
fix(clippy): remove duplicate arc_with_non_send_sync attribute in wal.rs
2025-08-24 22:59:47 +05:30
Pekka Enberg
c428ff06b2
sqlite3: Implement sqlite3_bind_parameter_index()
2025-08-24 20:10:31 +03:00
Pekka Enberg
4fc7b94a6b
Merge 'sqlite3: Implement sqlite3_clear_bindings()' from Pekka Enberg
...
Closes #2759
2025-08-24 19:57:14 +03:00
Pekka Enberg
9d2f26bb04
sqlite3: Implement sqlite3_clear_bindings()
2025-08-24 19:33:18 +03:00
Pekka Enberg
5a8f281555
Merge 'sqlite3: Implement sqlite3_get_autocommit()' from Pekka Enberg
...
Closes #2758
2025-08-24 16:53:04 +03:00
Pekka Enberg
a8ab049a36
Merge 'Add support for AEGIS encryption algorithm' from Avinash Sajjanshetty
...
Depends on #2722
This builds upon the #2722 PR which let me configure the encryption
algorithm. So this adds AEGIS and uses it as default.
Note that choice of cipher at higher APIs is still not possible. I have
a follow up PR which updates the PRAGMAs
AEGIS is way too damn fast, here are some numbers:
```
* MACs:
aegis128x4-mac : 223.91 Gb/s
aegis128x2-mac : 270.87 Gb/s
aegis128l-mac : 229.35 Gb/s
sthash : 83.60 Gb/s
hmac-sha256 (boring): 27.46 Gb/s
blake3 : 21.41 Gb/s
* Encryption:
aegis128x4 : 104.19 Gb/s
aegis128x2 : 182.46 Gb/s
aegis128l : 181.62 Gb/s
aegis256x2 : 133.45 Gb/s
aegis256x4 : 125.23 Gb/s
aegis256 : 102.12 Gb/s
aes128-gcm (aes-gcm): 2.16 Gb/s
aes128-gcm (boring) : 63.25 Gb/s
aes256-gcm (aes-gcm): 1.70 Gb/s
aes256-gcm (boring) : 59.14 Gb/s
chacha20-poly1305 : 2.39 Gb/s
ascon128a : 5.84 Gb/s
```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #2742
2025-08-24 14:16:10 +03:00
Pekka Enberg
f5dfbfffc2
Merge branch 'main' into aegis
2025-08-24 14:16:01 +03:00
Pekka Enberg
ea2192c332
sqlite3: Implement sqlite3_get_autocommit()
2025-08-24 14:13:36 +03:00
Pekka Enberg
2c6fa76437
cargo fmt
2025-08-24 14:13:20 +03:00
Avinash Sajjanshetty
011f878158
make clippy bro happy
2025-08-24 16:21:06 +05:30
Avinash Sajjanshetty
77a4e96022
run encryption tests in CI
2025-08-24 16:15:13 +05:30
Avinash Sajjanshetty
a4b9c33b81
Use the new API to init cipher
2025-08-24 16:15:13 +05:30
Avinash Sajjanshetty
53f9c0dc7a
Add support for lord AEGIS, the fastest and the greatest
2025-08-24 16:15:11 +05:30
TcMits
3cd0ebe22f
clippy
2025-08-24 14:40:48 +07:00
TcMits
d24812373f
missing context for to_string
2025-08-24 14:37:29 +07:00
TcMits
9e4f3b41ef
correctly implement get_column_name
2025-08-24 14:07:46 +07:00
TcMits
22b6bad2c0
Merge branch 'main' into clean-parser-4
2025-08-24 13:15:05 +07:00
Pekka Enberg
73414663f8
Merge 'bindings/java: Implement batch operations for JDBC4Statement' from Kim Seon Woo
...
Implement the following methods for JDBC4Statement:
- addBatch
- clearBatch
- executeBatch
Closes #2754
2025-08-24 08:54:44 +03:00
Pekka Enberg
147740c95e
Merge 'reduce cloning Token in parser' from Lâm Hoàng Phúc
...
small changes make parser 3-6% faster
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #2745
2025-08-24 08:53:35 +03:00
Pekka Enberg
22c9cb6618
s/PerConnEncryptionContext/EncryptionContext/
2025-08-24 08:17:20 +03:00
Pekka Enberg
1b89273f10
Merge 'refactor encryption module and make it configurable' from Avinash Sajjanshetty
...
Previously, the encryption module had hardcoded a lot of things. This
refactor makes it slightly nice and makes it configurable.
Right now cipher algorithm is assumed and hardcoded, I will make that
configurable in the upcoming PR
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #2722
2025-08-24 08:16:28 +03:00
김선우
7057c97cfe
Remove .rustc_info.json
2025-08-24 10:25:14 +09:00
김선우
fa8896d9ee
Nit
2025-08-24 10:20:39 +09:00
김선우
9f6eb8bc92
Update verification of batch compatible statements using regex
2025-08-24 10:13:04 +09:00
김선우
bf1473dc08
Override JDBC4PreparedStatement to throw exception when calling addBatch method
2025-08-24 09:35:29 +09:00
김선우
346525e5f0
Update test
2025-08-24 09:25:59 +09:00
김선우
df41994ecc
Implement execute batch
2025-08-24 09:15:07 +09:00
Preston Thorpe
b113c497aa
Merge 'Replace a couple refcells for types that trivially impl Copy' from Preston Thorpe
...
Closes #2750
2025-08-23 16:33:07 -04:00
Preston Thorpe
dccd09049e
Merge 'fix: normalize quotes in update' from
...
fixes : #2744
```
turso> create table simple(x);
turso> insert into "simple"(x) values (1);
turso> select * from "simple";
┌───┐
│ x │
├───┤
│ 1 │
└───┘
turso> update "simple" set x = 3 where "simple"."x" = 1;
turso> select * from "simple";
┌───┐
│ x │
├───┤
│ 3 │
└───┘
turso>
```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Reviewed-by: Preston Thorpe <preston@turso.tech >
Closes #2747
2025-08-23 16:31:44 -04:00
PThorpe92
9a418f1d3e
Replace a couple refcells with cell in pager
2025-08-23 15:55:01 -04:00
Jussi Saurio
b4ee40dd3d
fix tests
2025-08-23 16:14:02 +03:00
Jussi Saurio
1d24925e21
Make fill_cell_payload() safe for async IO and cache spilling
...
Problems:
1. fill_cell_payload() is not re-entrant because it can yield IO
on allocating a new overflow page, resulting in losing some of the
input data.
2. fill_cell_payload() in its current form is not safe for cache spilling
because the previous overflow page in the chain of allocated overflow pages
can be evicted by a spill caused by the next overflow page allocation,
invalidating the page pointer and causing corruption.
3. fill_cell_payload() uses raw pointers and `unsafe` as a workaround from a previous time when we used to clone `WriteState`, resulting in hard-to-read code.
Solutions:
1. Introduce a new substate to the fill_cell_payload state machine to handle
re-entrancy wrt. allocating overflow pages.
2. Always pin the current overflow page so that it cannot be evicted during the
overflow chain construction. Also pin the regular page the overflow chain is
attached to, because it is immediately accessed after fill_cell_payload is done.
3. Remove all explicit usages of `unsafe` from `fill_cell_payload` (although our pager is ofc still extremely unsafe under the hood :] )
Note that solution 2 addresses a problem that arose in the development of page cache
spilling, which is not yet implemented, but will be soon.
Miscellania:
1. Renamed a bunch of variables to be clearer
2. Added more comments about what is happening in fill_cell_payload
2025-08-23 16:14:02 +03:00
TcMits
399f10fe9a
refactor parser fmt
2025-08-23 19:16:26 +07:00
themixednuts
80eca66be9
fix: normalize quotes in update
...
fixes : #2744
2025-08-23 03:17:03 -05:00
TcMits
fd63688ede
reduce cloning Token in parser
2025-08-23 15:07:32 +07:00
Pekka Enberg
52f66e6c60
Merge 'Add syntax highlighting for EXPLAIN and ANALYZE' from Alex Miller
...
I'm working on ANALYZE. I'm using EXPLAIN. The lack of highlighting
for them in the CLI annoyed me a bit.
I don't think there's any tests for this? I'm mostly at a "it seems to
work for me". I double checked that `EXPLAIN SELECT CASE 0 WHEN 0 THEN
0 ELSE 1` syntax highlights, to make sure I didn't break the longer
parsing (which I had).
Closes #2741
2025-08-23 10:51:31 +03:00
Pekka Enberg
dd56eb7d9b
Merge 'Add basic support for ANALYZE statement' from Alex Miller
...
This is part 1 of N implementing #656
This permits only `ANALYZE <table_name>` to work, and all other forms
fail with a parse error (as documented in the tests).
On SQLite, ANALYZE generates:
```
sqlite> CREATE TABLE sqlite_stat1(tbl,idx,stat);
sqlite> CREATE TABLE iiftest(a int, b int, c int); sqlite> EXPLAIN ANALYZE iiftest;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 21 0 0 Start at 21
1 Null 0 1 0 0 r[1]=NULL
2 OpenWrite 3 4 0 3 0 root=4 iDb=0; sqlite_stat1
3 Rewind 3 9 0 0
4 Column 3 0 2 0 r[2]= cursor 3 column 0
5 Ne 3 8 2 BINARY-8 81 if r[2]!=r[3] goto 8
6 Rowid 3 4 0 0 r[4]=sqlite_stat1.rowid
7 Delete 3 0 0 sqlite_stat1 2
8 Next 3 4 0 1
9 OpenWrite 0 4 0 3 0 root=4 iDb=0; sqlite_stat1
10 OpenRead 4 2 0 3 0 root=2 iDb=0; iiftest
11 String8 0 11 0 iiftest 0 r[11]='iiftest'; iiftest
12 Count 4 13 0 0 r[13]=count()
13 IfNot 13 18 0 0
14 Null 0 12 0 0 r[12]=NULL
15 MakeRecord 11 3 9 BBB 0 r[9]=mkrec(r[11..13])
16 NewRowid 0 5 0 0 r[5]=rowid
17 Insert 0 9 5 8 intkey=r[5] data=r[9]
18 LoadAnalysis 0 0 0 0
19 Expire 0 0 0 0
20 Halt 0 0 0 0
21 Transaction 0 1 9 0 1 usesStmtJournal=0
22 String8 0 3 0 iiftest 0 r[3]='iiftest'
23 Goto 0 1 0 0
```
Turso can now generate:
```
turso> create table sqlite_stat1(tbl,idx,stat);
turso> create table iiftest(a int, b int, c int);
turso> explain analyze iiftest;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 19 0 0 Start at 19
1 Null 0 1 0 0 r[1]=NULL
2 OpenWrite 0 2 0 0 root=2; iDb=0
3 Rewind 0 9 0 0 Rewind sqlite_stat1
4 Column 0 0 2 0 r[2]=sqlite_stat1.tbl
5 Ne 2 3 9 0 if r[2]!=r[3] goto 9
6 RowId 0 4 0 0 r[4]=sqlite_stat1.rowid
7 Delete 0 0 0 sqlite_stat1 0
8 Next 0 4 0 0
9 OpenWrite 1 2 0 0 root=2; iDb=0
10 OpenRead 2 3 0 0 =iiftest, root=3, iDb=0
11 String8 0 7 0 iiftest 0 r[7]='iiftest'
12 Count 2 9 0 0
13 IfNot 9 18 0 0 if !r[9] goto 18
14 Null 0 8 0 0 r[8]=NULL
15 MakeRecord 7 3 6 0 r[6]=mkrec(r[7..9])
16 NewRowid 1 5 0 0 r[5]=rowid
17 Insert 1 6 5 sqlite_stat1 0 intkey=r[5] data=r[6]
18 Halt 0 0 0 0
19 String8 0 3 0 iiftest 0 r[3]='iiftest'
20 Goto 0 1 0 0
```
Note the missing support for LoadAnalysis and Expire, but there's no
optimizer work done yet to leverage any gathered statistics yet anyway.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #2740
2025-08-23 10:50:57 +03:00
Alex Miller
4619890ffc
Add basic support for ANALYZE statement.
...
This permits only `ANALYZE <table_name>` to work, and all other forms
fail with a parse error (as documented in the tests).
On SQLite, ANALYZE generates:
sqlite> CREATE TABLE sqlite_stat1(tbl,idx,stat);
sqlite> CREATE TABLE iiftest(a int, b int, c int);
sqlite> EXPLAIN ANALYZE iiftest;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 21 0 0 Start at 21
1 Null 0 1 0 0 r[1]=NULL
2 OpenWrite 3 4 0 3 0 root=4 iDb=0; sqlite_stat1
3 Rewind 3 9 0 0
4 Column 3 0 2 0 r[2]= cursor 3 column 0
5 Ne 3 8 2 BINARY-8 81 if r[2]!=r[3] goto 8
6 Rowid 3 4 0 0 r[4]=sqlite_stat1.rowid
7 Delete 3 0 0 sqlite_stat1 2
8 Next 3 4 0 1
9 OpenWrite 0 4 0 3 0 root=4 iDb=0; sqlite_stat1
10 OpenRead 4 2 0 3 0 root=2 iDb=0; iiftest
11 String8 0 11 0 iiftest 0 r[11]='iiftest'; iiftest
12 Count 4 13 0 0 r[13]=count()
13 IfNot 13 18 0 0
14 Null 0 12 0 0 r[12]=NULL
15 MakeRecord 11 3 9 BBB 0 r[9]=mkrec(r[11..13])
16 NewRowid 0 5 0 0 r[5]=rowid
17 Insert 0 9 5 8 intkey=r[5] data=r[9]
18 LoadAnalysis 0 0 0 0
19 Expire 0 0 0 0
20 Halt 0 0 0 0
21 Transaction 0 1 9 0 1 usesStmtJournal=0
22 String8 0 3 0 iiftest 0 r[3]='iiftest'
23 Goto 0 1 0 0
Turso can now generate:
turso> create table sqlite_stat1(tbl,idx,stat);
turso> create table iiftest(a int, b int, c int);
turso> explain analyze iiftest;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 19 0 0 Start at 19
1 Null 0 1 0 0 r[1]=NULL
2 OpenWrite 0 2 0 0 root=2; iDb=0
3 Rewind 0 9 0 0 Rewind sqlite_stat1
4 Column 0 0 2 0 r[2]=sqlite_stat1.tbl
5 Ne 2 3 9 0 if r[2]!=r[3] goto 9
6 RowId 0 4 0 0 r[4]=sqlite_stat1.rowid
7 Delete 0 0 0 sqlite_stat1 0
8 Next 0 4 0 0
9 OpenWrite 1 2 0 0 root=2; iDb=0
10 OpenRead 2 3 0 0 =iiftest, root=3, iDb=0
11 String8 0 7 0 iiftest 0 r[7]='iiftest'
12 Count 2 9 0 0
13 IfNot 9 18 0 0 if !r[9] goto 18
14 Null 0 8 0 0 r[8]=NULL
15 MakeRecord 7 3 6 0 r[6]=mkrec(r[7..9])
16 NewRowid 1 5 0 0 r[5]=rowid
17 Insert 1 6 5 sqlite_stat1 0 intkey=r[5] data=r[6]
18 Halt 0 0 0 0
19 String8 0 3 0 iiftest 0 r[3]='iiftest'
20 Goto 0 1 0 0
Note the missing support for LoadAnalysis and Expire, but there's no
optimizer work done yet to leverage any gathered statistics yet anyway.
2025-08-22 23:18:53 -07:00
Alex Miller
327936fb51
Highlight the EXPLAIN keyword
2025-08-22 22:49:58 -07:00
Alex Miller
f1250a6c86
Add syntax highlighting for ANALYZE
2025-08-22 22:47:46 -07:00
Pekka Enberg
78295e3b4c
Merge 'wal-api: allow to mix frames insert with SQL execution' from Nikita Sivukhin
...
This PR make it possible to do 2 pretty crazy things with turso-db:
1. Now we can mix WAL frames inserts with SQL execution within same
transaction. This will allow sync engine to execute rebase of local
changes within atomically over main database file (the operation first
require us to push new frames to physically revert local changes and
then we need to replay local logical changes on top of the modified DB
state)
2. Under `conn_raw_api` Cargo feature turso-db now expose method which
allow caller to specify WAL file path. This dangerous capability exposed
for sync-engine which maintain 2 databases: main one and "revert"-DB
which shares same DB file but has it's own separate WAL. As sync-engine
has full control over checkpoint - it can guarantee that DB file will be
consistent with both main and "revert" DB WALs.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com >
Closes #2716
2025-08-22 15:41:43 +03:00
