aljaz/turso
1
0
Fork 0
mirror of https://github.com/aljazceru/turso.git synced 2026-01-06 17:54:20 +01:00
Code Issues Packages Projects Releases Wiki Activity
3,853 Commits 1 Branch 0 Tags
b6718471ceb600d4e6ca8a7babdb1fb67db43f45
Commit Graph

488 Commits

Author SHA1 Message Date
Jussi Saurio
3e42a62cd0 Add SeekLE/SeekLT operations to VDBE 2025-04-09 10:14:29 +03:00
Jussi Saurio
ed929d2a0a Merge 'Properly handle insertion of indexed columns' from Preston Thorpe 
```console
limbo> create index p on products(price);
limbo> explain insert into products (name,price) values ('huh',32);
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     17    0                    0   Start at 17
1     OpenWriteAsync     0     273   0                    0
2     OpenWriteAwait     0     0     0                    0
3     String8            0     3     0     huh            0   r[3]='huh'
4     Integer            32    4     0                    0   r[4]=32
5     OpenWriteAsync     1     297   0                    0
6     OpenWriteAwait     0     0     0                    0
7     NewRowId           0     1     0                    0
8     MakeRecord         2     3     5                    0   r[5]=mkrec(r[2..4])
9     InsertAsync        0     5     1                    0
10    InsertAwait        0     0     0                    0
11    Copy               4     6     0                    0   r[6]=r[4]
12    Copy               1     7     0                    0   r[7]=r[1]
13    MakeRecord         6     2     8                    0   r[8]=mkrec(r[6..7])
14    IdxInsertAsync     1     8     6                    0   key=r[8]
15    IdxInsertAwait     1     0     0                    0
16    Halt               0     0     0                    0
17    Transaction        0     1     0                    0   write=true
18    Null               0     2     0                    0   r[2]=NULL
19    Goto               0     1     0                    0
```
When an insert happens, we need to be sure to insert into the index
btree as well.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1265
 2025-04-09 10:13:59 +03:00
PThorpe92
4b9b6c969b Parse schema rows after extensions are loaded 2025-04-08 20:10:47 -04:00
PThorpe92
1f29307fe8 Support proper index handling when doing insertions 2025-04-08 08:55:14 -04:00
PThorpe92
a2b9d8d371 Use Correct flag on idx insert to prevent seeking 2025-04-05 11:19:09 -04:00
Pekka Enberg
c592e27dca core/vdbe: Move explain() to last method in Program 
...it's the least interesting one and we should have `step()` at the top.
 2025-04-02 18:55:35 +03:00
Pere Diaz Bou
7e4b57f2e2 VDBE with direct function dispatch 
This PR is unapologetically stolen from @vmg's implementation in Vitess
implemented here https://github.com/vitessio/vitess/pull/12369. If you
want a more in depth explanation of how this works you can read the
[blog post he carefully
wrote](https://planetscale.com/blog/faster-interpreters-in-go-catching-up-with-cpp).

In limbo we have a huge problem with [register
spilling](https://en.wikipedia.org/wiki/Register_allocation), this can
be easily observed with the prolog of `Program::step` before:
```llvm
start:
    %e.i.i304.i = alloca [0 x i8], align 8
    %formatter.i305.i = alloca [64 x i8], align 8
    %buf.i306.i = alloca [24 x i8], align 8
    %formatter.i259.i = alloca [64 x i8], align 8
    ..................... these are repeated for hundreds of lines
.....................
    %formatter.i52.i = alloca [64 x i8], align 8
    %buf.i53.i = alloca [24 x i8], align 8
    %formatter.i.i = alloca [64 x i8], align 8
    %buf.i.i = alloca [24 x i8], align 8
    %_87.i = alloca [48 x i8], align 8
    %_82.i = alloca [24 x i8], align 8
    %_73.i = alloca [24 x i8], align 8
    %_66.i8446 = alloca [24 x i8], align 8
    %_57.i = alloca [24 x i8], align 8
    %_48.i = alloca [24 x i8], align 8
```

After these changes we completely remove the need of register spilling
(yes that is the complete prolog):
```llvm
start:
    %self1 = alloca [80 x i8], align 8
    %pager = alloca [8 x i8], align 8
    %mv_store = alloca [8 x i8], align 8
    store ptr %0, ptr %mv_store, align 8
    store ptr %1, ptr %pager, align 8
    %2 = getelementptr inbounds i8, ptr %state, i64 580
    %3 = getelementptr inbounds i8, ptr %state, i64 576
    %4 = getelementptr inbounds i8, ptr %self, i64 16
    %5 = getelementptr inbounds i8, ptr %self, i64 8
    %6 = getelementptr inbounds i8, ptr %self1, i64 8
    br label %bb1, !dbg !286780
```
When it comes to branch prediction, we don't really fix a lot because
thankfully rust already compiles `match` expressions
to a jump table:

```llvm
%insn = getelementptr inbounds [0 x %"vdbe::insn::Insn"], ptr %self657,
i64 0, i64 %index, !dbg !249527
%332 = load i8, ptr %insn, align 8, !dbg !249528, !range !38291,
!noundef !14
switch i8 %332, label %default.unreachable26674 [
    i8 0, label %bb111
    i8 1, label %bb101
    i8 2, label %bb100
    i8 3, label %bb110
    ...
    i8 104, label %bb5
    i8 105, label %bb16
    i8 106, label %bb14
], !dbg !249530
```

Some results
----
```
function dispatch:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [29.498 ns 29.548 ns 29.601 ns]
                        change: [-3.6125% -3.3592% -3.0804%] (p = 0.00 <
0.05)

main:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [33.789 ns 33.832 ns 33.878 ns]
```
 2025-04-02 14:55:37 +02:00
Pekka Enberg
6199c3994a Merge 'Create plan for Update queries' from Preston Thorpe 
closes #1186, or at least works towards it by implementing an actual
Plan for update queries instead of translating everything inline. This
allows for actually using index's, rewriting const expressions, pushing
predicates instead of hardcoding a full scan in the translation.
### TODOs:
1.  `RETURNING` clause/result columns
2.  `OFFSET` clauses
3. on conflict
### LIMIT:
By supporting `LIMIT` directly in update queries, we'll have to put the
tests outside of the compatibility tests, maybe in the CLI tests.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1189
 2025-04-01 17:33:32 +03:00
Pekka Enberg
169864456e Merge 'Fix IdxCmp insn comparisons' from Jussi Saurio 
We never hit bugs due to these because of 1. not having multi column
indexes in our TCL test databases, 2. otherwise not really having Rust
tests involving indexes, and 3. `IdxLt` and `IdxLe` not actually being
used anywhere yet
Also as @PThorpe92 pointed out there are some nuances to the comparison
logic we may need to eventually implement regarding comparisons with
uneven number of keys:
https://github.com/sqlite/sqlite/blob/master/src/vdbeaux.c#L4719

Reviewed-by: Preston Thorpe (@PThorpe92)

Closes #1215
 2025-03-31 10:40:01 +03:00
Jussi Saurio
42e25d23dd Fix IdxCmp insn comparisons 2025-03-30 23:01:41 +03:00
PThorpe92
516e443a2b Fix use index cursor id in emitter and prevent reinsert pk on update 2025-03-30 12:15:25 -04:00
Ihor Andrianov
a234aa3647 remove vec cloning from json agg functions 2025-03-30 19:10:15 +03:00
Ihor Andrianov
40bb867d54 clippy 2025-03-30 19:01:16 +03:00
Ihor Andrianov
db5e364210 made json an optional module again 2025-03-30 19:01:03 +03:00
Ihor Andrianov
6c126dcd97 add jsonb_set 2025-03-30 18:58:40 +03:00
Ihor Andrianov
101dd51d7c add jsonb_group_object and array 2025-03-30 18:58:39 +03:00
Ihor Andrianov
568dc54b9e big cleanup 2025-03-30 18:58:33 +03:00
Ihor Andrianov
a983c979c6 jsonb_merge, json_group_array, json_group_object 2025-03-30 18:47:33 +03:00
Pere Diaz Bou
a13b33fec9 clippy again 2025-03-29 22:07:43 +01:00
Pere Diaz Bou
d9f5cd870d clippy 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
4a9c4cff02 fix comparison of immutable records in seekgt 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
1bfec65f23 remove dbg 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
ee55116ca6 return row as reference to registers 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
5b7fcd27bd make column reuse blob/text fields 2025-03-29 22:02:49 +01:00
Pere Diaz Bou
bf37fd3314 wip 2025-03-29 22:02:49 +01:00
Pekka Enberg
4ee60348f2 Merge 'Fixes probably all floating point math issues and floating point display issues.' from Ihor Andrianov 
Closes #1206
Closes #447
Closes #1117
Issues:
 1. Rust floating point math fucntions are non deterministic.
 2. SQLite have complex floating point display rules.
I changed rust functions to libm for math ops and implemented Display
trait for OwnedValue:Float. A lot of float formatting SQLite probably
inherits from C have to be handcrafted.

Closes #1208
 2025-03-29 17:33:53 +02:00
Ihor Andrianov
8b9f34af71 fix tests and return nan as null 2025-03-29 14:46:11 +02:00
Ihor Andrianov
303f1b3749 replace std math functions with libm for compat 2025-03-29 12:16:14 +02:00
Pere Diaz Bou
83d0c9a1b6 fix read overflow page procedure 2025-03-28 11:30:19 +01:00
Pere Diaz Bou
dc8acf1a4a cell_get no allocations 2025-03-28 11:12:27 +01:00
Pekka Enberg
387b68fc06 Merge 'Expose 'Explain' to prepared statement to allow for alternate Writer ' from Preston Thorpe 
### The problem:
I often need to copy the output of an `Explain` statement to my
clipboard. Currently this is not possible because it currently will only
write to stdout.
All other limbo output, I am able to run `.output file` in the CLI, then
enter my query and in another tmux pane I simply `cat file | xclip -in
-selection clipboard`.
### The solution:
Expose a `statement.explain()` method that returns the query explanation
as a string. If the user uses something like `execute` instead of
prepare, it will default to `stdout` as expected, but this allows the
user to access the query plan on the prepared statement and do with it
what they please.

Closes #1166
 2025-03-28 09:55:58 +02:00
Pere Diaz Bou
9291f60722 Introduce Register struct 
OwnedValue has become a powerhouse of madness, mainly because I decided
to do it like that when I first introduced AggContext. I decided it was
enough and I introduced a `Register` struct that contains `OwnedValue`,
`Record` and `Aggregation`, this way we don't use `OwnedValue` for
everything make everyone's life harder.

This is the next step towards making ImmutableRecords the default
because I want to remove unnecessary allocations. Right now we clone
OwnedValues when we generate a record more than needed.
 2025-03-27 17:53:02 +01:00
Pere Diaz Bou
8642d416c7 Introduce immutable record. 
Currently we have a Record, which is a dumb vector of cloned values.
This is incredibly bad for performance as we do not want to clone
objects unless needed. Therefore, let's start by introducing this type
so that any record that has already been serialized will be returned
from btree in the format of a simple payload with reference to payload.
 2025-03-25 17:35:41 +01:00
PThorpe92
7b55f7a167 Move explain to statement to allow for alternate writer 2025-03-24 18:48:12 -04:00
Ihor Andrianov
d8e070a360 moved json_cache to state 2025-03-24 14:48:40 +02:00
Ihor Andrianov
1511c9b3bf add json cache to json functions and fix tests 2025-03-24 13:17:58 +02:00
Pekka Enberg
65bf33023c core: Fix Destroy opcode root page handling 
The `p1` register points to the root page, not to a cursor.

Fixes #1136
 2025-03-24 10:54:49 +02:00
Pekka Enberg
31bbc5144a Merge 'Initial pass at UPDATE support' from Preston Thorpe 
This PR is to support `Update` queries. Follows sqlite behavior as much
as possible.
### limbo
```console
limbo> create table t (a,b,c);
limbo> explain update t set a = 1 where b = 2;
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     18    0                    0   Start at 18
1     OpenWriteAsync     0     2     0                    0
2     OpenWriteAwait     0     0     0                    0
3     RewindAsync        0     0     0                    0
4     RewindAwait        0     17    0                    0   Rewind table t
5       Column           0     1     4                    0   r[4]=t.b
6       Ne               4     5     15                   0   if r[4]!=r[5] goto 15
7       RowId            0     6     0                    0   r[6]=t.rowid
8       IsNull           6     17    0                    0   if (r[6]==NULL) goto 17
9       Integer          1     1     0                    0   r[1]=1
10      Column           0     1     2                    0   r[2]=t.b
11      Column           0     2     3                    0   r[3]=t.c
12      MakeRecord       1     3     7                    0   r[7]=mkrec(r[1..3])
13      InsertAsync      0     7     6                    0
14      InsertAwait      0     0     0                    0
15    NextAsync          0     0     0                    0
16    NextAwait          0     5     0                    0
17    Halt               0     0     0                    0
18    Transaction        0     1     0                    0   write=true
19    Integer            2     5     0                    0   r[5]=2
20    Goto               0     1     0                    0
```
### sqlite
```console
sqlite> explain update t set a = 1 where b = 2;
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     15    0                    0   Start at 15
1     Null           0     1     2                    0   r[1..2]=NULL
2     Noop           1     0     1                    0
3     OpenWrite      0     2     0     2              0   root=2 iDb=0; t
4     Rewind         0     14    0                    0
5       Column         0     1     5                    0   r[5]= cursor 0 column 1
6       Ne             6     13    5     BINARY-8       81  if r[5]!=r[6] goto 13
7       Rowid          0     2     0                    0   r[2]= rowid of 0
8       IsNull         2     14    0                    0   if r[2]==NULL goto 14
9       Integer        1     3     0                    0   r[3]=1
10      Column         0     1     4                    0   r[4]= cursor 0 column 1
11      MakeRecord     3     2     1                    0   r[1]=mkrec(r[3..4])
12      Insert         0     1     2     t              7   intkey=r[2] data=r[1]
13    Next           0     5     0                    1
14    Halt           0     0     0                    0
15    Transaction    0     1     1     0              1   usesStmtJournal=0
16    Integer        2     6     0                    0   r[6]=2
17    Goto           0     1     0                    0
```

Closes #1130
 2025-03-24 09:19:22 +02:00
Pekka Enberg
e8c0a6e728 Merge 'Various JSON improvements' from Ihor Andrianov 
Added jsonb_object, jsonb_array, json_insert, jsonb_insert.
MongoDB is sweating now.

Closes #1160
 2025-03-24 09:17:40 +02:00
PThorpe92
c83cc6dff2 Small nits/clippy errors in vdbe 2025-03-23 17:08:15 -04:00
Ihor Andrianov
2cab36bfc3 add json_replace, jsonb_replace 2025-03-23 20:52:03 +02:00
Ihor Andrianov
c4549ad2cd split json traversal and mutation operation logic 2025-03-23 20:37:12 +02:00
Ihor Andrianov
252583cb1e add jsonb_obj jsonb_array 2025-03-20 15:22:52 +02:00
Pekka Enberg
4142f4f4cb Merge 'Organize extension library and feature gate VFS' from Preston Thorpe 
I keep having 3+ PR's in at the same time and always deal with crazy
conflicts because everything in the `ext` library is together in one
file.
This PR moves each category of extension into its own file, and
separates the `vfs` functionality in Core into the `ext/dynamic` module,
so that it can be more easily separated from wasm (or non feature =
"fs") targets to prevent build issues.
The only semantic changes made in this PR is the feature gating of vfs,
the rest is simply organizing and cleaning up imports.
Was unsure if `vfs` should be a feature on the `core` side too, or to
just enable it with the `fs` feature which seemed reasonable, as that
was already the current behavior. But let me know if we want it entirely
behind it's own feature.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1124
 2025-03-19 19:08:13 +02:00
Pekka Enberg
d4db5eb4c1 Merge 'Various JSON and JSONB function improvements' from Ihor Andrianov 
Added jsonb_remove, jsonb_replace, json_replace.
Updated json_remove to use jsonb under the hood.
Fixed json function big numbers serialization.
Add tests for new functions.

Closes #1140
 2025-03-19 17:21:44 +02:00
PThorpe92
57d4aa7216 Reorganize ext library and feature gate vfs to more easily prevent wasm build issues 2025-03-19 10:17:11 -04:00
Ihor Andrianov
32ea972151 make tests pass 2025-03-19 11:29:46 +02:00
Ihor Andrianov
b5e86a9e36 remove and replace functions defenitions 2025-03-18 21:43:48 +02:00
Pekka Enberg
a2c6831f30 Merge 'Implement FastLock for DatabaseHeader' from Pere Diaz Bou 
The motivation behind implementing our own lock is to not depend on any
dependency as we should moving forward. This is a experiment for now as
a single test obviously is not enough but I believe this is the right
direction to on.
## benchmark tldr;
Execute benchmarks have a performance improvement around [1.78%, 7.5%]
which seems like it went okay as it was expected from removing
`pthread_mutex` calls.
## benchmarks before
```
Prepare `SELECT 1`/Limbo/SELECT 1
                        time:   [575.63 ns 577.33 ns 580.07 ns]
                        change: [-1.3304% -0.8881% -0.4675%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Prepare `SELECT * FROM users LIMIT 1`/Limbo/SELECT * FROM users LIMIT 1
                        time:   [1.2070 µs 1.2114 µs 1.2166 µs]
                        change: [-0.8670% -0.4084% -0.0252%] (p = 0.06 > 0.05)
                        No change in performance detected.

Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou...
                        time:   [2.9845 µs 2.9895 µs 2.9951 µs]
                        change: [-3.0470% -2.6038% -2.1301%] (p = 0.00 < 0.05)
                        Performance has improved.
Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou... #2
                        time:   [1.6015 µs 1.6084 µs 1.6157 µs]
                        change: [-0.0676% +0.3850% +0.8704%] (p = 0.11 > 0.05)
                        No change in performance detected.
Execute `SELECT * FROM users LIMIT ?`/Limbo/1
                        time:   [442.46 ns 446.72 ns 454.13 ns]
                        change: [+3.9744% +4.5337% +5.3357%] (p = 0.00 < 0.05)
                        Performance has regressed.
Execute `SELECT * FROM users LIMIT ?`/Limbo/10
                        time:   [3.1722 µs 3.1850 µs 3.1980 µs]
                        change: [+7.1994% +7.7452% +8.2856%] (p = 0.00 < 0.05)
                        Performance has regressed.
Execute `SELECT * FROM users LIMIT ?`/Limbo/50
                        time:   [14.976 µs 15.024 µs 15.078 µs]
                        change: [+5.7879% +6.2419% +6.7139%] (p = 0.00 < 0.05)
                        Performance has regressed.
Execute `SELECT * FROM users LIMIT ?`/Limbo/100
                        time:   [29.834 µs 29.925 µs 30.024 µs]
                        change: [+4.6519% +5.0384% +5.4491%] (p = 0.00 < 0.05)
                        Performance has regressed.
Execute `SELECT 1`/Limbo
                        time:   [45.135 ns 45.439 ns 45.763 ns]
                        change: [-0.4703% -0.0496% +0.3622%] (p = 0.81 > 0.05)
                        No change in performance detected.
```
## benchmarks after
```
Prepare `SELECT 1`/Limbo/SELECT 1
                        time:   [585.61 ns 590.92 ns 596.49 ns]
                        change: [+0.5902% +1.1505% +1.7012%] (p = 0.00 < 0.05)
                        Change within noise threshold.

Prepare `SELECT * FROM users LIMIT 1`/Limbo/SELECT * FROM users LIMIT 1
                        time:   [1.2061 µs 1.2090 µs 1.2119 µs]
                        change: [-0.2364% +0.0977% +0.4252%] (p = 0.57 > 0.05)
                        No change in performance detected.

Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou...
                        time:   [2.9854 µs 2.9893 µs 2.9936 µs]
                        change: [-0.5752% -0.2529% +0.0167%] (p = 0.09 > 0.05)
                        No change in performance detected.
Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou... #2
                        time:   [1.5853 µs 1.5983 µs 1.6108 µs]
                        change: [-2.3810% -1.7986% -1.2748%] (p = 0.00 < 0.05)
                        Performance has improved.
Execute `SELECT * FROM users LIMIT ?`/Limbo/1
                        time:   [429.84 ns 431.34 ns 433.07 ns]
                        change: [-2.7721% -1.8504% -0.8738%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Execute `SELECT * FROM users LIMIT ?`/Limbo/10
                        time:   [2.9184 µs 2.9254 µs 2.9323 µs]
                        change: [-8.2377% -7.7816% -7.3373%] (p = 0.00 < 0.05)
                        Performance has improved.
Execute `SELECT * FROM users LIMIT ?`/Limbo/50
                        time:   [14.190 µs 14.229 µs 14.271 µs]
                        change: [-6.2034% -5.7858% -5.3552%] (p = 0.00 < 0.05)
                        Performance has improved.
Execute `SELECT * FROM users LIMIT ?`/Limbo/100
                        time:   [28.734 µs 28.856 µs 28.979 µs]
                        change: [-4.3640% -3.9462% -3.5492%] (p = 0.00 < 0.05)
                        Performance has improved.

Execute `SELECT 1`/Limbo
                        time:   [43.144 ns 43.237 ns 43.326 ns]
                        change: [-4.9417% -4.5554% -4.2030%] (p = 0.00 < 0.05)
                        Performance has improved.
```

Closes #1120
 2025-03-18 13:44:16 +02:00
Pekka Enberg
f9d7834874 Merge 'Jsonb extract' from Ihor Andrianov 
Made a jsonb traversal by json path.
Changed some ordinary json functions to use jsonb under the hood, so now
behavior of our json module more like sqlite.
Found and fixed some bugs on the way.

Closes #1135
 2025-03-17 18:25:28 +02:00