aljaz/turso
1
0
Fork 0
mirror of https://github.com/aljazceru/turso.git synced 2026-01-05 01:04:22 +01:00
Code Issues Packages Projects Releases Wiki Activity
8,035 Commits 1 Branch 0 Tags
370da9fa590e73759d7ac5dc6464b59b92ec6cfa
Commit Graph

4410 Commits

Author SHA1 Message Date
Alex Miller
370da9fa59 ANALYZE creates sqlite_stat1 if it doesn't exist 
This change replaces a bail_parse_error!() when sqlite_stat1 doesn't
exist with the appropriate codegen to create the table, and handle both
cases of the table existing or not existing.

SQLite's codegen looks like:

sqlite> create table stat_test(a,b,c);
sqlite> explain analyze stat_test;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     40    0                    0   Start at 40
1     ReadCookie     0     3     2                    0
2     If             3     5     0                    0
3     SetCookie      0     2     4                    0
4     SetCookie      0     5     1                    0
5     CreateBtree    0     2     1                    0   r[2]=root iDb=0 flags=1
6     OpenWrite      0     1     0     5              0   root=1 iDb=0
7     NewRowid       0     1     0                    0   r[1]=rowid
8     Blob           6     3     0                   0   r[3]= (len=6)
9     Insert         0     3     1                    8   intkey=r[1] data=r[3]
10    Close          0     0     0                    0
11    Close          0     0     0                    0
12    Null           0     4     5                    0   r[4..5]=NULL
13    Noop           4     0     4                    0
14    OpenWrite      3     1     0     5              0   root=1 iDb=0; sqlite_master
15    SeekRowid      3     17    1                    0   intkey=r[1]
16    Rowid          3     5     0                    0   r[5]= rowid of 3
17    IsNull         5     26    0                    0   if r[5]==NULL goto 26
18    String8        0     6     0     table          0   r[6]='table'
19    String8        0     7     0     sqlite_stat1   0   r[7]='sqlite_stat1'
20    String8        0     8     0     sqlite_stat1   0   r[8]='sqlite_stat1'
21    Copy           2     9     0                    0   r[9]=r[2]
22    String8        0     10    0     CREATE TABLE sqlite_stat1(tbl,idx,stat) 0   r[10]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
23    MakeRecord     6     5     4     BBBDB          0   r[4]=mkrec(r[6..10])
24    Delete         3     68    5                    0
25    Insert         3     4     5                    0   intkey=r[5] data=r[4]
26    SetCookie      0     1     2                    0
27    ParseSchema    0     0     0     tbl_name='sqlite_stat1' AND type!='trigger' 0
28    OpenWrite      0     2     0     3              16  root=2 iDb=0; sqlite_stat1
29    OpenRead       5     2     0     3              0   root=2 iDb=0; stat_test
30    String8        0     18    0     stat_test      0   r[18]='stat_test'; stat_test
31    Count          5     20    0                    0   r[20]=count()
32    IfNot          20    37    0                    0
33    Null           0     19    0                    0   r[19]=NULL
34    MakeRecord     18    3     16    BBB            0   r[16]=mkrec(r[18..20])
35    NewRowid       0     12    0                    0   r[12]=rowid
36    Insert         0     16    12                   8   intkey=r[12] data=r[16]
37    LoadAnalysis   0     0     0                    0
38    Expire         0     0     0                    0
39    Halt           0     0     0                    0
40    Transaction    0     1     1     0              1   usesStmtJournal=1
41    Goto           0     1     0                    0

And now Turso's looks like:

turso> create table stat_test(a,b,c);
turso> explain analyze stat_test;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     23    0                    0   Start at 23
1     Null               0     1     0                    0   r[1]=NULL
2     CreateBtree        0     2     1                    0   r[2]=root iDb=0 flags=1
3     OpenWrite          0     1     0                    0   root=1; iDb=0
4     NewRowid           0     3     0                    0   r[3]=rowid
5     String8            0     4     0     table          0   r[4]='table'
6     String8            0     5     0     sqlite_stat1   0   r[5]='sqlite_stat1'
7     String8            0     6     0     sqlite_stat1   0   r[6]='sqlite_stat1'
8     Copy               2     7     0                    0   r[7]=r[2]
9     String8            0     8     0     CREATE TABLE sqlite_stat1(tbl,idx,stat)  0   r[8]='CREATE TABLE sqlite_stat1(tbl,idx,stat)'
10    MakeRecord         4     5     9                    0   r[9]=mkrec(r[4..8])
11    Insert             0     9     3     sqlite_stat1   0   intkey=r[3] data=r[9]
12    ParseSchema        0     0     0     tbl_name = 'sqlite_stat1' AND type != 'trigger'  0   tbl_name = 'sqlite_stat1' AND type != 'trigger'
13    OpenWrite          1     2     0                    0   root=2; iDb=0
14    OpenRead           2     2     0                    0   =stat_test, root=2, iDb=0
15    String8            0     12    0     stat_test      0   r[12]='stat_test'
16    Count              2     14    0                    0
17    IfNot              14    22    0                    0   if !r[14] goto 22
18    Null               0     13    0                    0   r[13]=NULL
19    MakeRecord         12    3     11                   0   r[11]=mkrec(r[12..14])
20    NewRowid           1     10    0                    0   r[10]=rowid
21    Insert             1     11    10    sqlite_stat1   0   intkey=r[10] data=r[11]
22    Halt               0     0     0                    0
23    Goto               0     1     0                    0

The notable difference in size is following the same codegen difference
in CREATE TABLE, where sqlite's odd dance of adding a placeholder entry
which is immediately replaced is instead done in tursodb as just
inserting the correct row in the first place. Aside from lines 6-13 of
sqlite's vdbe being missing, there's still the lack of LoadAnalysis,
Expire, and Cookie management.
 2025-08-24 13:35:39 -07:00
Pekka Enberg
de16c770c3 Merge 'Remove duplicated attribute in ' from bit-aloo 
Removed the redundant `#![allow(clippy::arc_with_non_send_sync)]` from
wal.rs, since it’s already defined in `mod.rs`.

Closes #2764
 2025-08-24 20:54:26 +03:00
bit-aloo
37cebb0669 fix(clippy): remove duplicate arc_with_non_send_sync attribute in wal.rs 2025-08-24 22:59:47 +05:30
Pekka Enberg
c428ff06b2 sqlite3: Implement sqlite3_bind_parameter_index() 2025-08-24 20:10:31 +03:00
Pekka Enberg
9d2f26bb04 sqlite3: Implement sqlite3_clear_bindings() 2025-08-24 19:33:18 +03:00
Avinash Sajjanshetty
011f878158 make clippy bro happy 2025-08-24 16:21:06 +05:30
Avinash Sajjanshetty
a4b9c33b81 Use the new API to init cipher 2025-08-24 16:15:13 +05:30
Avinash Sajjanshetty
53f9c0dc7a Add support for lord AEGIS, the fastest and the greatest 2025-08-24 16:15:11 +05:30
Pekka Enberg
22c9cb6618 s/PerConnEncryptionContext/EncryptionContext/ 2025-08-24 08:17:20 +03:00
Pekka Enberg
1b89273f10 Merge 'refactor encryption module and make it configurable' from Avinash Sajjanshetty 
Previously, the encryption module had hardcoded a lot of things. This
refactor makes it slightly nice and makes it configurable.
Right now cipher algorithm is assumed and hardcoded, I will make that
configurable in the upcoming PR

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2722
 2025-08-24 08:16:28 +03:00
Preston Thorpe
b113c497aa Merge 'Replace a couple refcells for types that trivially impl Copy' from Preston Thorpe 
Closes #2750
 2025-08-23 16:33:07 -04:00
PThorpe92
9a418f1d3e Replace a couple refcells with cell in pager 2025-08-23 15:55:01 -04:00
themixednuts
80eca66be9 fix: normalize quotes in update 
fixes: #2744
 2025-08-23 03:17:03 -05:00
Alex Miller
4619890ffc Add basic support for ANALYZE statement. 
This permits only `ANALYZE <table_name>` to work, and all other forms
fail with a parse error (as documented in the tests).

On SQLite, ANALYZE generates:

sqlite> CREATE TABLE sqlite_stat1(tbl,idx,stat);
sqlite> CREATE TABLE iiftest(a int, b int, c int);
sqlite> EXPLAIN ANALYZE iiftest;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     21    0                    0   Start at 21
1     Null           0     1     0                    0   r[1]=NULL
2     OpenWrite      3     4     0     3              0   root=4 iDb=0; sqlite_stat1
3     Rewind         3     9     0                    0
4       Column         3     0     2                    0   r[2]= cursor 3 column 0
5       Ne             3     8     2     BINARY-8       81  if r[2]!=r[3] goto 8
6       Rowid          3     4     0                    0   r[4]=sqlite_stat1.rowid
7       Delete         3     0     0     sqlite_stat1   2
8     Next           3     4     0                    1
9     OpenWrite      0     4     0     3              0   root=4 iDb=0; sqlite_stat1
10    OpenRead       4     2     0     3              0   root=2 iDb=0; iiftest
11    String8        0     11    0     iiftest        0   r[11]='iiftest'; iiftest
12    Count          4     13    0                    0   r[13]=count()
13    IfNot          13    18    0                    0
14    Null           0     12    0                    0   r[12]=NULL
15    MakeRecord     11    3     9     BBB            0   r[9]=mkrec(r[11..13])
16    NewRowid       0     5     0                    0   r[5]=rowid
17    Insert         0     9     5                    8   intkey=r[5] data=r[9]
18    LoadAnalysis   0     0     0                    0
19    Expire         0     0     0                    0
20    Halt           0     0     0                    0
21    Transaction    0     1     9     0              1   usesStmtJournal=0
22    String8        0     3     0     iiftest        0   r[3]='iiftest'
23    Goto           0     1     0                    0

Turso can now generate:

turso> create table sqlite_stat1(tbl,idx,stat);
turso> create table iiftest(a int, b int, c int);
turso> explain analyze iiftest;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     19    0                    0   Start at 19
1     Null               0     1     0                    0   r[1]=NULL
2     OpenWrite          0     2     0                    0   root=2; iDb=0
3     Rewind             0     9     0                    0   Rewind  sqlite_stat1
4       Column           0     0     2                    0   r[2]=sqlite_stat1.tbl
5       Ne               2     3     9                    0   if r[2]!=r[3] goto 9
6       RowId            0     4     0                    0   r[4]=sqlite_stat1.rowid
7       Delete           0     0     0     sqlite_stat1   0
8     Next               0     4     0                    0
9     OpenWrite          1     2     0                    0   root=2; iDb=0
10    OpenRead           2     3     0                    0   =iiftest, root=3, iDb=0
11    String8            0     7     0     iiftest        0   r[7]='iiftest'
12    Count              2     9     0                    0
13    IfNot              9     18    0                    0   if !r[9] goto 18
14    Null               0     8     0                    0   r[8]=NULL
15    MakeRecord         7     3     6                    0   r[6]=mkrec(r[7..9])
16    NewRowid           1     5     0                    0   r[5]=rowid
17    Insert             1     6     5     sqlite_stat1   0   intkey=r[5] data=r[6]
18    Halt               0     0     0                    0
19    String8            0     3     0     iiftest        0   r[3]='iiftest'
20    Goto               0     1     0                    0

Note the missing support for LoadAnalysis and Expire, but there's no
optimizer work done yet to leverage any gathered statistics yet anyway.
 2025-08-22 23:18:53 -07:00
Pekka Enberg
78295e3b4c Merge 'wal-api: allow to mix frames insert with SQL execution' from Nikita Sivukhin 
This PR make it possible to do 2 pretty crazy things with turso-db:
1. Now we can mix WAL frames inserts with SQL execution within same
transaction. This will allow sync engine to execute rebase of local
changes within atomically over main database file (the operation first
require us to push new frames to physically revert local changes and
then we need to replay local logical changes on top of the modified DB
state)
2. Under `conn_raw_api` Cargo feature turso-db now expose method which
allow caller to specify WAL file path. This dangerous capability exposed
for sync-engine which maintain 2 databases: main one and "revert"-DB
which shares same DB file but has it's own separate WAL. As sync-engine
has full control over checkpoint - it can guarantee that DB file will be
consistent with both main and "revert" DB WALs.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2716
 2025-08-22 15:41:43 +03:00
Jussi Saurio
14873c76fb unixio: use Mutex::lock() instead of Mutex::try_lock() 
we should wait to obtain the lock, not immediately fail if we cant.
 2025-08-22 10:47:50 +03:00
Pekka Enberg
b9bb859271 Merge 'Switch to new parser in core' from Levy A. 
Integrate #2381 to core. Resolves #2337.

Reviewed-by: Lâm Hoàng Phúc (@TcMits)

Closes #2650
 2025-08-22 10:06:37 +03:00
Pekka Enberg
156352fabc Merge ' should not return a Completion when there is a page cache hit' from Pedro Muniz 
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2723
 2025-08-22 07:33:28 +03:00
Jussi Saurio
4edbf3aac0 Merge 'Page cache truncate' from Nikita Sivukhin 
Add `truncate` method in the page cache which remove all entries which
reference pages greater than new DB size.
This will be used in the sync engine as in its case DB size can shrink
when we "rebase" changes from remote to local.
It stands on the #2707 because touch few files from that PR

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2711
 2025-08-21 23:34:39 +03:00
Levy A.
ee12ef9fb5 remove unnecessary Box<ast::Select> 2025-08-21 17:20:25 -03:00
Levy A.
07975603d3 fix: incorrect sql statement in parser test 2025-08-21 15:24:01 -03:00
Levy A.
4ba1304fb9 complete parser integration 2025-08-21 15:23:59 -03:00
Levy A.
186e2f5d8e switch to new parser 2025-08-21 15:19:16 -03:00
pedrocarlo
6b0ed08465 read_page should return No Completion when have a page cache hit 2025-08-21 14:39:24 -03:00
Avinash Sajjanshetty
3090545167 use encryption ctx instead of encryption key 2025-08-21 22:36:32 +05:30
Avinash Sajjanshetty
cc8c763942 refactor encryption module and make it configurable 2025-08-21 22:36:31 +05:30
Pekka Enberg
ae8b1eb00d Merge 'core/io: Don't open file as non-blocking in Unix backend' from Pekka Enberg 
The Unix backend is a syscall()-based, blocking implementation. The
O_NONBLOCK adds nothing.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2708
 2025-08-21 19:13:39 +03:00
Nikita Sivukhin
94ed92ca4e fix compilation and clippy 2025-08-21 18:28:16 +04:00
Nikita Sivukhin
4653f78356 fix compilation 2025-08-21 18:28:16 +04:00
Nikita Sivukhin
7720e07931 allow to mix frames push with sql execution in the single WAL session 
- we will use this in the sync engine
 2025-08-21 18:28:16 +04:00
Nikita Sivukhin
732d998618 allow to open DB with different WAL file :mindblow: 2025-08-21 18:28:16 +04:00
Nikita Sivukhin
89e180c2a8 add few tests 2025-08-21 18:27:54 +04:00
Nikita Sivukhin
94f2e5a4b9 add truncate method to the page cache 2025-08-21 18:27:54 +04:00
Jussi Saurio
05f4cec34d Merge 'Wal api checkpoint seq' from Nikita Sivukhin 
This PR adds information about checkpoint sequence number to the WAL raw
API. Will be used in the sync engine.
Depends on the #2699

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2707
 2025-08-21 17:05:45 +03:00
Jussi Saurio
cc28b8833e Fix condition that checks table.cols against number of provided values 2025-08-21 16:40:10 +03:00
Jussi Saurio
b5bd31a47b Remove old unused data structures and functions 2025-08-21 16:40:10 +03:00
Jussi Saurio
ac56d5bb67 Use new datastructures and functions in translate_insert 2025-08-21 16:40:10 +03:00
Jussi Saurio
88c4eae63e Add functions for constructing and translating Insertions 2025-08-21 16:40:10 +03:00
Jussi Saurio
630441e270 Add new Insertion datastructures 2025-08-21 16:40:10 +03:00
Jussi Saurio
215485d403 Add Table::get_column_by_name method 2025-08-21 16:40:10 +03:00
Jussi Saurio
dd2e0ea596 Fix: always emit rowid when column is rowid alias 
SQLite does not store the rowid alias column in the record at all
when it is a rowid alias, because the rowid is always stored anyway
in the record header.
 2025-08-21 16:40:10 +03:00
Jussi Saurio
f9ad43a3a3 Merge 'Fix: all indexes need to be updated if the rowid changes' from Jussi Saurio 
Found when running simulator in #2641
All indexes store the rowid as the last column, so whenever the rowid of
a given row changes the index entry must also be deleted and reinserted
with the new index.

Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2712
 2025-08-21 16:40:03 +03:00
Nikita Sivukhin
d7e47c1268 fix bug - continue checkpoint as usual even if frames range is degenerate 2025-08-21 17:37:19 +04:00
Jussi Saurio
e224bb15a8 Fix incorrect UNIQUE constraint failure behavior in UPDATE 
UPDATE should skip over the UNIQUE constraint failure if the existing
row it found during the check has the same rowid as the row we are
currently updating
 2025-08-21 16:30:34 +03:00
Jussi Saurio
e7fbeba120 Merge 'Fix: in UPDATE, insert rowid into index instead of NULL' from Jussi Saurio 
Same deal as #2700, except this time in UPDATE. Nothing tests this on
`main` so not caught.
I will later put #2641 into mergeable condition so it will catch all of
these going forward.

Reviewed-by: Nikita Sivukhin (@sivukhin)

Closes #2710
 2025-08-21 15:49:55 +03:00
Jussi Saurio
9d44e97a7a Fix: all indexes need to be updated if the rowid changes 2025-08-21 15:48:46 +03:00
Jussi Saurio
4ceadcca78 Merge 'Remove hardcoded flag usage in DBHeader for encryption' from Avinash Sajjanshetty 
Previously, we just hardcoded the reserved space with encryption flag.
This patch removes that and sets the reserved space if a key was
specified during a creation of db

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2706
 2025-08-21 15:46:41 +03:00
Jussi Saurio
d1647390c4 Merge 'add remove_file method to the IO' from Nikita Sivukhin 
Closes #2705
 2025-08-21 15:46:16 +03:00
Jussi Saurio
bec2c4bc79 Merge 'properly execute pragmas - they may require some IO' from Nikita Sivukhin 
- for example CDC pragma execute create table under the hood

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2701
 2025-08-21 15:45:49 +03:00
Jussi Saurio
6c0c4d77d0 Fix UPDATE inserting NULL into index instead of rowid 2025-08-21 14:58:25 +03:00