Commit Graph

563 Commits

Author SHA1 Message Date
Pekka Enberg
2d3fd01f91 Merge 'Support Create Index' from Preston Thorpe
Closes #1193
```console
│limbo> explain create index idxp on products(price);
│addr  opcode             p1    p2    p3    p4             p5  comment
│----  -----------------  ----  ----  ----  -------------  --  -------
│0     Init               0     39    0                    0   Start at 39
│1     CreateBtree        0     1     2                    0   r[1]=root iDb=0 flags=2
│2     OpenWriteAsync     0     1     0                    0
│3     OpenWriteAwait     0     0     0                    0
│4     NewRowId           0     2     0                    0
│5     String8            0     3     0     index          0   r[3]='index'
│6     String8            0     4     0     idxp           0   r[4]='idxp'
│7     String8            0     5     0     products       0   r[5]='products'
│8     Copy               1     6     1                    0   r[6]=r[1]
│9     String8            0     7     0     CREATE INDEX idxp ON products (price)  0   r[7]='CREATE INDEX idxp ON products (price)'
│10    MakeRecord         3     5     8                    0   r[8]=mkrec(r[3..7])
│11    InsertAsync        0     8     2                    0
│12    InsertAwait        0     0     0                    0
│13    SorterOpen         3     1     0     k(1,B)         0   cursor=3
│14    OpenPseudo         4     9     2                    0   2 columns in r[9]
│15    OpenReadAsync      2     273   0                    0   table=products, root=273
│16    OpenReadAwait      0     0     0                    0
│17    RewindAsync        2     0     0                    0
│18    RewindAwait        2     25    0                    0   Rewind table products
│19      Column           2     2     10                   0   r[10]=products.price
│20      RowId            2     11    0                    0   r[11]=products.rowid
│21      MakeRecord       10    2     12                   0   r[12]=mkrec(r[10..11])
│22      SorterInsert     3     12    0     0              0   key=r[12]
│23    NextAsync          2     0     0                    0
│24    NextAwait          2     19    0                    0
│25    OpenWriteAsync     1     1     0                    0
│26    OpenWriteAwait     0     0     0                    0
│27    SorterSort         3     33    0                    0
│28      SorterData       3     13    4                    0   r[13]=data
│29      SeekEnd          1     0     0                    0
│30      IdxInsertAsync   1     13    0                    0   key=r[13]
│31      IdxInsertAwait   1     0     0                    0
│32    SorterNext         3     28    0                    0
│33    Close              3     0     0                    0
│34    Close              2     0     0                    0
│35    Close              1     0     0                    0
│36    ParseSchema        0     0     0     name = 'idxp' AND type = 'index'  0   name = 'idxp' AND type = 'index'
│37    Close              0     0     0                    0
│38    Halt               0     0     0                    0
│39    Transaction        0     1     0                    0   write=true
│40    Goto               0     1     0                    0
```
This will create the initial index btree and insert whatever relevant
records that need to be inserted, it doesn't handle the case of
inserting new index keys when normal records are created afterwards.
That will prob be added in next PR to keep this one concise.
Limbo will properly use the index in a subsequent query:
![image](https://github.com/user-
attachments/assets/eb41e985-4a70-49a5-8218-62c25e4d16c5)
Creating a unique index on a column that has 2 existing identical rows:
![image](https://github.com/user-
attachments/assets/ea46c720-5235-4451-81f0-25497ed9ee92)

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1199
2025-04-06 19:32:36 +03:00
PThorpe92
67eda10453 Allow reading altered tables by defaulting to null in Column insn 2025-04-05 16:19:56 -04:00
PThorpe92
83af71f140 Return accidentally deleted comment on SeekGE insn from merge conflict 2025-04-05 11:30:57 -04:00
PThorpe92
a2b9d8d371 Use Correct flag on idx insert to prevent seeking 2025-04-05 11:19:09 -04:00
PThorpe92
068ab4ab27 Refactor btree to reuse existing insert and seek with idx keys 2025-04-05 11:19:09 -04:00
PThorpe92
e020ba3dfe Add enum for interpreting a value as a register or literal for insns 2025-04-05 11:19:07 -04:00
PThorpe92
b0016a0ee2 Support create index with SeekEnd and IdxCreate opcode functionality 2025-04-05 11:15:36 -04:00
PThorpe92
45a8e5e226 Add close_cursors helper method to program builder 2025-04-05 11:06:18 -04:00
Pekka Enberg
c3eaf47180 Merge 'Add support for default values in INSERT statements' from Diego Reis
While working on #494 I noticed that default values defined in schemas
weren't being applied.
Before:
![image](https://github.com/user-attachments/assets/af3a31be-
accd-47b4-848f-3820497e32ac)
Now:
![image](https://github.com/user-
attachments/assets/0ccb194c-4caa-481d-8f47-2405c3db06bb)

Closes #1249
2025-04-04 08:59:44 +03:00
Diego Reis
38d842d675 docs/insn: Standardizes comments for insn to doc comments 2025-04-04 00:56:54 -03:00
PThorpe92
ae2be30204 Move init label to proper place in create vtab translation 2025-04-03 20:22:14 -04:00
Pekka Enberg
24063bd9c0 core/vdbe: Add newline between op functions 2025-04-02 18:57:07 +03:00
Pekka Enberg
c592e27dca core/vdbe: Move explain() to last method in Program
...it's the least interesting one and we should have `step()` at the top.
2025-04-02 18:55:35 +03:00
Pekka Enberg
3420955db7 core/vdbe: Rename execute_insn_* to op_*
The "execute::execute_insn" prefix is noisy, let's rename the
instruction operation functions to something shorter and sweeter.
2025-04-02 18:02:02 +03:00
Pere Diaz Bou
66f70d571d fmt 2025-04-02 13:14:26 +00:00
Pere Diaz Bou
f5221589f0 remove wrong usage of feature = json 2025-04-02 15:00:51 +02:00
Pere Diaz Bou
7e4b57f2e2 VDBE with direct function dispatch
This PR is unapologetically stolen from @vmg's implementation in Vitess
implemented here https://github.com/vitessio/vitess/pull/12369. If you
want a more in depth explanation of how this works you can read the
[blog post he carefully
wrote](https://planetscale.com/blog/faster-interpreters-in-go-catching-up-with-cpp).

In limbo we have a huge problem with [register
spilling](https://en.wikipedia.org/wiki/Register_allocation), this can
be easily observed with the prolog of `Program::step` before:
```llvm
start:
    %e.i.i304.i = alloca [0 x i8], align 8
    %formatter.i305.i = alloca [64 x i8], align 8
    %buf.i306.i = alloca [24 x i8], align 8
    %formatter.i259.i = alloca [64 x i8], align 8
    ..................... these are repeated for hundreds of lines
.....................
    %formatter.i52.i = alloca [64 x i8], align 8
    %buf.i53.i = alloca [24 x i8], align 8
    %formatter.i.i = alloca [64 x i8], align 8
    %buf.i.i = alloca [24 x i8], align 8
    %_87.i = alloca [48 x i8], align 8
    %_82.i = alloca [24 x i8], align 8
    %_73.i = alloca [24 x i8], align 8
    %_66.i8446 = alloca [24 x i8], align 8
    %_57.i = alloca [24 x i8], align 8
    %_48.i = alloca [24 x i8], align 8
```

After these changes we completely remove the need of register spilling
(yes that is the complete prolog):
```llvm
start:
    %self1 = alloca [80 x i8], align 8
    %pager = alloca [8 x i8], align 8
    %mv_store = alloca [8 x i8], align 8
    store ptr %0, ptr %mv_store, align 8
    store ptr %1, ptr %pager, align 8
    %2 = getelementptr inbounds i8, ptr %state, i64 580
    %3 = getelementptr inbounds i8, ptr %state, i64 576
    %4 = getelementptr inbounds i8, ptr %self, i64 16
    %5 = getelementptr inbounds i8, ptr %self, i64 8
    %6 = getelementptr inbounds i8, ptr %self1, i64 8
    br label %bb1, !dbg !286780
```
When it comes to branch prediction, we don't really fix a lot because
thankfully rust already compiles `match` expressions
to a jump table:

```llvm
%insn = getelementptr inbounds [0 x %"vdbe::insn::Insn"], ptr %self657,
i64 0, i64 %index, !dbg !249527
%332 = load i8, ptr %insn, align 8, !dbg !249528, !range !38291,
!noundef !14
switch i8 %332, label %default.unreachable26674 [
    i8 0, label %bb111
    i8 1, label %bb101
    i8 2, label %bb100
    i8 3, label %bb110
    ...
    i8 104, label %bb5
    i8 105, label %bb16
    i8 106, label %bb14
], !dbg !249530
```

Some results
----
```
function dispatch:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [29.498 ns 29.548 ns 29.601 ns]
                        change: [-3.6125% -3.3592% -3.0804%] (p = 0.00 <
0.05)

main:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [33.789 ns 33.832 ns 33.878 ns]
```
2025-04-02 14:55:37 +02:00
Pekka Enberg
6199c3994a Merge 'Create plan for Update queries' from Preston Thorpe
closes #1186, or at least works towards it by implementing an actual
Plan for update queries instead of translating everything inline. This
allows for actually using index's, rewriting const expressions, pushing
predicates instead of hardcoding a full scan in the translation.
### TODOs:
1.  `RETURNING` clause/result columns
2.  `OFFSET` clauses
3. on conflict
### LIMIT:
By supporting `LIMIT` directly in update queries, we'll have to put the
tests outside of the compatibility tests, maybe in the CLI tests.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1189
2025-04-01 17:33:32 +03:00
Pekka Enberg
4c93c69e5a Merge 'Let remainder (%) accept textual arguments' from Anton Harniakou
Also I added more tests for exec_add, exec_subtract, exec_multiply,
exec_divide, exec_remainder.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1214
2025-03-31 10:40:21 +03:00
Pekka Enberg
169864456e Merge 'Fix IdxCmp insn comparisons' from Jussi Saurio
We never hit bugs due to these because of 1. not having multi column
indexes in our TCL test databases, 2. otherwise not really having Rust
tests involving indexes, and 3. `IdxLt` and `IdxLe` not actually being
used anywhere yet
Also as @PThorpe92 pointed out there are some nuances to the comparison
logic we may need to eventually implement regarding comparisons with
uneven number of keys:
https://github.com/sqlite/sqlite/blob/master/src/vdbeaux.c#L4719

Reviewed-by: Preston Thorpe (@PThorpe92)

Closes #1215
2025-03-31 10:40:01 +03:00
Jussi Saurio
42e25d23dd Fix IdxCmp insn comparisons 2025-03-30 23:01:41 +03:00
PThorpe92
516e443a2b Fix use index cursor id in emitter and prevent reinsert pk on update 2025-03-30 12:15:25 -04:00
Ihor Andrianov
a234aa3647 remove vec cloning from json agg functions 2025-03-30 19:10:15 +03:00
Ihor Andrianov
40bb867d54 clippy 2025-03-30 19:01:16 +03:00
Ihor Andrianov
db5e364210 made json an optional module again 2025-03-30 19:01:03 +03:00
Ihor Andrianov
6c126dcd97 add jsonb_set 2025-03-30 18:58:40 +03:00
Ihor Andrianov
101dd51d7c add jsonb_group_object and array 2025-03-30 18:58:39 +03:00
Ihor Andrianov
568dc54b9e big cleanup 2025-03-30 18:58:33 +03:00
Ihor Andrianov
a983c979c6 jsonb_merge, json_group_array, json_group_object 2025-03-30 18:47:33 +03:00
Anton Harniakou
9a61c75ea1 Let remainder (%) accept textual arguments; add more tests for core/vdbe/insn.rs
Tests for exec_add, exec_subtract, exec_multiply, exec_divide,
exec_remainder
2025-03-30 12:14:05 +03:00
Pere Diaz Bou
a13b33fec9 clippy again 2025-03-29 22:07:43 +01:00
Pere Diaz Bou
d9f5cd870d clippy 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
4a9c4cff02 fix comparison of immutable records in seekgt 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
1bfec65f23 remove dbg 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
ee55116ca6 return row as reference to registers 2025-03-29 22:04:08 +01:00
Pere Diaz Bou
5b7fcd27bd make column reuse blob/text fields 2025-03-29 22:02:49 +01:00
Pere Diaz Bou
bf37fd3314 wip 2025-03-29 22:02:49 +01:00
Pekka Enberg
4ee60348f2 Merge 'Fixes probably all floating point math issues and floating point display issues.' from Ihor Andrianov
Closes #1206
Closes #447
Closes #1117
Issues:
 1. Rust floating point math fucntions are non deterministic.
 2. SQLite have complex floating point display rules.
I changed rust functions to libm for math ops and implemented Display
trait for OwnedValue:Float. A lot of float formatting SQLite probably
inherits from C have to be handcrafted.

Closes #1208
2025-03-29 17:33:53 +02:00
Ihor Andrianov
09b55a9b41 fix nan to return null 2025-03-29 14:47:08 +02:00
Ihor Andrianov
8b9f34af71 fix tests and return nan as null 2025-03-29 14:46:11 +02:00
Ihor Andrianov
303f1b3749 replace std math functions with libm for compat 2025-03-29 12:16:14 +02:00
Pere Diaz Bou
83d0c9a1b6 fix read overflow page procedure 2025-03-28 11:30:19 +01:00
Pere Diaz Bou
dc8acf1a4a cell_get no allocations 2025-03-28 11:12:27 +01:00
Pekka Enberg
387b68fc06 Merge 'Expose 'Explain' to prepared statement to allow for alternate Writer ' from Preston Thorpe
### The problem:
I often need to copy the output of an `Explain` statement to my
clipboard. Currently this is not possible because it currently will only
write to stdout.
All other limbo output, I am able to run `.output file` in the CLI, then
enter my query and in another tmux pane I simply `cat file | xclip -in
-selection clipboard`.
### The solution:
Expose a `statement.explain()` method that returns the query explanation
as a string. If the user uses something like `execute` instead of
prepare, it will default to `stdout` as expected, but this allows the
user to access the query plan on the prepared statement and do with it
what they please.

Closes #1166
2025-03-28 09:55:58 +02:00
Pere Diaz Bou
9291f60722 Introduce Register struct
OwnedValue has become a powerhouse of madness, mainly because I decided
to do it like that when I first introduced AggContext. I decided it was
enough and I introduced a `Register` struct that contains `OwnedValue`,
`Record` and `Aggregation`, this way we don't use `OwnedValue` for
everything make everyone's life harder.

This is the next step towards making ImmutableRecords the default
because I want to remove unnecessary allocations. Right now we clone
OwnedValues when we generate a record more than needed.
2025-03-27 17:53:02 +01:00
Pekka Enberg
c1a0236dcc Merge 'Introduce immutable record' from Pere Diaz Bou
Currently we have a Record, which is a dumb vector of cloned values.
This is incredibly bad for performance as we do not want to clone
objects unless needed. Therefore, let's start by introducing this type
so that any record that has already been serialized will be returned
from btree in the format of a simple payload with reference to payload.

Closes #1176
2025-03-26 20:31:04 +02:00
Pekka Enberg
fca643641c Merge 'Fix compute_shl negate with overflow' from Krishna Vishal
Fixes #1156
- Changed `compute_shl` implementation to handle negation with overflow.
- Added TCL tests.

Closes #1171
2025-03-26 16:08:59 +02:00
Pere Diaz Bou
8642d416c7 Introduce immutable record.
Currently we have a Record, which is a dumb vector of cloned values.
This is incredibly bad for performance as we do not want to clone
objects unless needed. Therefore, let's start by introducing this type
so that any record that has already been serialized will be returned
from btree in the format of a simple payload with reference to payload.
2025-03-25 17:35:41 +01:00
krishvishal
785be8479f Fix a fuzzer failure and add tcl test covering the failure 2025-03-25 11:43:51 +05:30
krishvishal
b55dc586bd change compute_shl implementation to handle negation with overflow 2025-03-25 10:10:15 +05:30