Commit Graph

3667 Commits

Author SHA1 Message Date
Pere Diaz Bou
e2d00a9f96 inline start transactions from pager and wal
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [30.543 ns 30.585 ns 30.632 ns]
2025-04-02 16:18:36 +02:00
Pere Diaz Bou
2a49fe9bd2 Remove RWLock from Shared wal state
WalShared state can be shared without having to wrap everything with a
lock, and instead use atomics on some places and rwlock on others -- for
now.

## Results:
From:
----
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [34.125 ns 34.218 ns 34.324 ns]
Execute `SELECT 1`/sqlite_execute_select_1
                        time:   [28.124 ns 28.254 ns 28.385 ns]

To:
----
Gnuplot not found, using plotters backend
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [31.919 ns 32.113 ns 32.327 ns]
Execute `SELECT 1`/sqlite_execute_select_1
                        time:   [29.662 ns 29.900 ns 30.139 ns]
2025-04-02 16:18:36 +02:00
Pekka Enberg
cd5ef7c7db Merge 'Reuse register in binary expressions if they're equal ' from Diego Reis
Alongside with #1227, this PR closes #1226
Before:
![image](https://github.com/user-attachments/assets/d9fd4ff4-957d-46cd-
abe0-63d0b2c4acdb)
After:
![image](https://github.com/user-
attachments/assets/2ce4b8f1-f8f8-423a-8d35-d947de2794cb)

Closes #1234
2025-04-02 17:02:35 +03:00
Pekka Enberg
0d3e92d42f Merge 'VDBE with indirect function dispatch' from Pere Diaz Bou
This PR is unapologetically stolen from @vmg's implementation in Vitess
implemented here https://github.com/vitessio/vitess/pull/12369. If you
want a more in depth explanation of how this works you can read the
[blog post he carefully
wrote](https://planetscale.com/blog/faster-interpreters-in-go-catching-
up-with-cpp).
In limbo we have a huge problem with [register
spilling](https://en.wikipedia.org/wiki/Register_allocation), this can
be easily observed with the prolog of `Program::step` before:
```llvm
start:
    %e.i.i304.i = alloca [0 x i8], align 8
    %formatter.i305.i = alloca [64 x i8], align 8
    %buf.i306.i = alloca [24 x i8], align 8
    %formatter.i259.i = alloca [64 x i8], align 8
    ..................... these are repeated for hundreds of lines
.....................
    %formatter.i52.i = alloca [64 x i8], align 8
    %buf.i53.i = alloca [24 x i8], align 8
    %formatter.i.i = alloca [64 x i8], align 8
    %buf.i.i = alloca [24 x i8], align 8
    %_87.i = alloca [48 x i8], align 8
    %_82.i = alloca [24 x i8], align 8
    %_73.i = alloca [24 x i8], align 8
    %_66.i8446 = alloca [24 x i8], align 8
    %_57.i = alloca [24 x i8], align 8
    %_48.i = alloca [24 x i8], align 8
```
After these changes we completely remove the need of register spilling
(yes that is the complete prolog):
```llvm
start:
    %self1 = alloca [80 x i8], align 8
    %pager = alloca [8 x i8], align 8
    %mv_store = alloca [8 x i8], align 8
    store ptr %0, ptr %mv_store, align 8
    store ptr %1, ptr %pager, align 8
    %2 = getelementptr inbounds i8, ptr %state, i64 580
    %3 = getelementptr inbounds i8, ptr %state, i64 576
    %4 = getelementptr inbounds i8, ptr %self, i64 16
    %5 = getelementptr inbounds i8, ptr %self, i64 8
    %6 = getelementptr inbounds i8, ptr %self1, i64 8
    br label %bb1, !dbg !286780
```
When it comes to branch prediction, we don't really fix a lot because
thankfully rust already compiles `match` expressions to a jump table:
```llvm
%insn = getelementptr inbounds [0 x %"vdbe::insn::Insn"], ptr %self657,
i64 0, i64 %index, !dbg !249527
%332 = load i8, ptr %insn, align 8, !dbg !249528, !range !38291,
!noundef !14
switch i8 %332, label %default.unreachable26674 [
    i8 0, label %bb111
    i8 1, label %bb101
    i8 2, label %bb100
    i8 3, label %bb110
    ...
    i8 104, label %bb5
    i8 105, label %bb16
    i8 106, label %bb14
], !dbg !249530
```
Some results
----
```
function dispatch:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [29.498 ns 29.548 ns 29.601 ns]
                        change: [-3.6125% -3.3592% -3.0804%] (p = 0.00 <
0.05)

main:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [33.789 ns 33.832 ns 33.878 ns]
```

Closes #1233
2025-04-02 17:01:58 +03:00
Pere Diaz Bou
66f70d571d fmt 2025-04-02 13:14:26 +00:00
Pere Diaz Bou
f5221589f0 remove wrong usage of feature = json 2025-04-02 15:00:51 +02:00
Pere Diaz Bou
7e4b57f2e2 VDBE with direct function dispatch
This PR is unapologetically stolen from @vmg's implementation in Vitess
implemented here https://github.com/vitessio/vitess/pull/12369. If you
want a more in depth explanation of how this works you can read the
[blog post he carefully
wrote](https://planetscale.com/blog/faster-interpreters-in-go-catching-up-with-cpp).

In limbo we have a huge problem with [register
spilling](https://en.wikipedia.org/wiki/Register_allocation), this can
be easily observed with the prolog of `Program::step` before:
```llvm
start:
    %e.i.i304.i = alloca [0 x i8], align 8
    %formatter.i305.i = alloca [64 x i8], align 8
    %buf.i306.i = alloca [24 x i8], align 8
    %formatter.i259.i = alloca [64 x i8], align 8
    ..................... these are repeated for hundreds of lines
.....................
    %formatter.i52.i = alloca [64 x i8], align 8
    %buf.i53.i = alloca [24 x i8], align 8
    %formatter.i.i = alloca [64 x i8], align 8
    %buf.i.i = alloca [24 x i8], align 8
    %_87.i = alloca [48 x i8], align 8
    %_82.i = alloca [24 x i8], align 8
    %_73.i = alloca [24 x i8], align 8
    %_66.i8446 = alloca [24 x i8], align 8
    %_57.i = alloca [24 x i8], align 8
    %_48.i = alloca [24 x i8], align 8
```

After these changes we completely remove the need of register spilling
(yes that is the complete prolog):
```llvm
start:
    %self1 = alloca [80 x i8], align 8
    %pager = alloca [8 x i8], align 8
    %mv_store = alloca [8 x i8], align 8
    store ptr %0, ptr %mv_store, align 8
    store ptr %1, ptr %pager, align 8
    %2 = getelementptr inbounds i8, ptr %state, i64 580
    %3 = getelementptr inbounds i8, ptr %state, i64 576
    %4 = getelementptr inbounds i8, ptr %self, i64 16
    %5 = getelementptr inbounds i8, ptr %self, i64 8
    %6 = getelementptr inbounds i8, ptr %self1, i64 8
    br label %bb1, !dbg !286780
```
When it comes to branch prediction, we don't really fix a lot because
thankfully rust already compiles `match` expressions
to a jump table:

```llvm
%insn = getelementptr inbounds [0 x %"vdbe::insn::Insn"], ptr %self657,
i64 0, i64 %index, !dbg !249527
%332 = load i8, ptr %insn, align 8, !dbg !249528, !range !38291,
!noundef !14
switch i8 %332, label %default.unreachable26674 [
    i8 0, label %bb111
    i8 1, label %bb101
    i8 2, label %bb100
    i8 3, label %bb110
    ...
    i8 104, label %bb5
    i8 105, label %bb16
    i8 106, label %bb14
], !dbg !249530
```

Some results
----
```
function dispatch:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [29.498 ns 29.548 ns 29.601 ns]
                        change: [-3.6125% -3.3592% -3.0804%] (p = 0.00 <
0.05)

main:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [33.789 ns 33.832 ns 33.878 ns]
```
2025-04-02 14:55:37 +02:00
Diego Reis
3c531ac5ec core/expr: Reuse register in binary expressions if they're equal 2025-04-02 09:15:41 -03:00
Diego Reis
86f8719b69 core/expr: Extract binary insn emission in a separate function 2025-04-02 09:14:01 -03:00
Pekka Enberg
65ae698773 Limbo 0.0.18 2025-04-02 15:04:48 +03:00
Pekka Enberg
f7d76e0fab Update CHANGELOG.md 2025-04-02 15:04:39 +03:00
Pekka Enberg
e79da7375b Limbo 0.0.18-pre.5 2025-04-02 13:38:22 +03:00
Pekka Enberg
7394ad6854 Disable more b-tree fuzzers... 2025-04-02 13:38:09 +03:00
Pekka Enberg
d9562a3d82 Add update-version.py script
Simplifies version bumping.
2025-04-02 09:31:28 +03:00
Pekka Enberg
f74a10c9c1 Limbo 0.0.18-pre.4 2025-04-02 09:30:42 +03:00
Pekka Enberg
cc8340d30e Disable btree_insert_fuzz_run_random test 2025-04-02 09:15:01 +03:00
Pekka Enberg
6199c3994a Merge 'Create plan for Update queries' from Preston Thorpe
closes #1186, or at least works towards it by implementing an actual
Plan for update queries instead of translating everything inline. This
allows for actually using index's, rewriting const expressions, pushing
predicates instead of hardcoding a full scan in the translation.
### TODOs:
1.  `RETURNING` clause/result columns
2.  `OFFSET` clauses
3. on conflict
### LIMIT:
By supporting `LIMIT` directly in update queries, we'll have to put the
tests outside of the compatibility tests, maybe in the CLI tests.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1189
2025-04-01 17:33:32 +03:00
Pekka Enberg
149f4aa729 Merge 'Remove unnecessary balance code that crashes' from Pere Diaz Bou
Closes #1223
2025-04-01 17:31:23 +03:00
Pekka Enberg
33b123bc2a Merge 'Validate cells inside a page after each operation' from Pere Diaz Bou
We need to ensure an operation doesn't transform the cells inside a page
to an invalid state. In debug mode we can enable a
`debug_validate_cells` with `#[cfg(debug_assertions)]` so that it is
skipped on release mode. This is good to catch things early instead of
seeing the page in an incredibly weird state some state in the future.

Closes #1222
2025-04-01 17:31:08 +03:00
Pere Diaz Bou
6b05dbddb0 remove unnecessary code while building count old and size old balancing 2025-04-01 13:11:55 +02:00
Pere Diaz Bou
70c5cf3970 Merge 'Refactor Cli Repl Commands to use clap' from Pedro Muniz
This PR changes the argument parsing and matching to use Clap instead of
our own handrolled one. This makes it much easier to add new commands
and modify existing ones by using the full power of clap's derive
macros. It also produces nice error and help messages for us. However,
there is a bug in Clap that is not correctly modifying the `display
name` in the help section. So the command would appear as `show` instead
of `.show` in the help messages. This is very minimal, but if this is a
blocker for this PR we can just overwrite the help message in its
entirety. Also using Clap would enable us to use its autocomplete crate
to generate autocompletions for these special repl commands which would
be a huge win when compared to the `sqlite3` cli.
This is the current help message:
```sh

Limbo SQL Shell Help
==============
Welcome to the Limbo SQL Shell! You can execute any standard SQL command here.
In addition to standard SQL commands, the following special commands are available:

Usage: <COMMAND>

Commands:
  exit       Exit this program with return-code CODE
  quit       Quit the shell
  open       Open a database file
  schema     Print this message or the help of the given subcommand(s) Display schema for a table
  output     Set output file (or stdout if empty)
  mode       Set output display mode
  opcodes    Show vdbe opcodes
  cd         Change the current working directory
  show       Display information about settings
  nullvalue  Set the value of NULL to be printed in 'list' mode
  echo       Toggle 'echo' mode to repeat commands before execution
  tables     Display tables
  import     Import data from FILE into TABLE
  load       Loads an extension library
  dump       Dump the current database as a list of SQL statements
  listvfs    List vfs modules available
  help       Print this message or the help of the given subcommand(s)

Usage Examples:
---------------
1. To quit the Limbo SQL Shell:
   .quit

2. To open a database file at path './employees.db':
   .open employees.db

3. To view the schema of a table named 'employees':
   .schema employees

4. To list all tables:
   .tables

5. To list all available SQL opcodes:
   .opcodes

6. To change the current output mode to 'pretty':
   .mode pretty

7. Send output to STDOUT if no file is specified:
   .output

8. To change the current working directory to '/tmp':
   .cd /tmp

9. Show the current values of settings:
   .show

10. To import csv file 'sample.csv' into 'csv_table' table:
   .import --csv sample.csv csv_table

11. To display the database contents as SQL:
   .dump

12. To load an extension library:
   .load /target/debug/liblimbo_regexp

13. To list all available VFS:
   .listvfs

Note:
- All SQL commands must end with a semicolon (;).
- Special commands start with a dot (.) and are not required to end with a semicolon.

```
If we need more information on a specific command, we can leverage CLAP
and do for instance:
```sh
.open -h
```

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1110
2025-04-01 11:40:14 +02:00
Pere Diaz Bou
141303e330 Validate cells inside a page after each operation
We need to ensure an operation doesn't transform the cells inside a page
to an invalid state. In debug mode we can enable a
`debug_validate_cells` with `#[cfg(debug_assertions)]` so that it is
skipped on release mode.

Modify pager logs
2025-04-01 11:19:23 +02:00
Pekka Enberg
5f90385a59 Merge 'Fix propagation of divider cell balancing interior page ' from Pere Diaz Bou
Newly added divider cells to parent of an interior page must point to
the page in question. Moreover rightmost pointer of the page will point
to previous divider cell pointer.

Closes #1221
2025-04-01 09:39:39 +03:00
Pere Diaz Bou
bab748e538 fix key generation 2025-04-01 01:05:07 +02:00
Pere Diaz Bou
d2642dfe0c skip repeated keys 2025-04-01 00:58:10 +02:00
Pere Diaz Bou
4308f8c73a Fix propagation of divider cell balancing interior page
Newly added divider cells to parent of an interior page must point to
the page in question. Moreover rightmost pointer of the page will point
to previous divider cell pointer.
2025-04-01 00:58:10 +02:00
Pere Diaz Bou
6c8061f8d7 Merge 'Allow balance_root to balance with interior pages' from Pere Diaz Bou
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1218
2025-04-01 00:50:23 +02:00
Pere Diaz Bou
24e4af7ee8 Allow balance_root to balance with interior pages 2025-03-31 12:42:01 +02:00
Pekka Enberg
1137678531 Merge 'Fuzz test btree fix seeking.' from Pere Diaz Bou
`seek` can return I/O but we never looped until it finished. I also
added test to ensure we don't repeat keys for now.

Closes #1217
2025-03-31 12:06:54 +03:00
Pere Diaz Bou
78f6480e8f remove ignored from fuzz tests 2025-03-31 10:59:28 +02:00
Pere Diaz Bou
bc660446a8 fuzz test ensure we "seek" until done 2025-03-31 10:57:55 +02:00
Pere Diaz Bou
0653ccf711 ensure btree fuzz doesn't repeat keys for now 2025-03-31 10:57:30 +02:00
Pekka Enberg
4c93c69e5a Merge 'Let remainder (%) accept textual arguments' from Anton Harniakou
Also I added more tests for exec_add, exec_subtract, exec_multiply,
exec_divide, exec_remainder.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1214
2025-03-31 10:40:21 +03:00
Pekka Enberg
77e9737b92 Merge 'JSON code cleanups' from Pekka Enberg
Just some renames to avoid redundant "json_" prefix and minor
conditional compilation cleanups.

Closes #1216
2025-03-31 10:40:09 +03:00
Pekka Enberg
169864456e Merge 'Fix IdxCmp insn comparisons' from Jussi Saurio
We never hit bugs due to these because of 1. not having multi column
indexes in our TCL test databases, 2. otherwise not really having Rust
tests involving indexes, and 3. `IdxLt` and `IdxLe` not actually being
used anywhere yet
Also as @PThorpe92 pointed out there are some nuances to the comparison
logic we may need to eventually implement regarding comparisons with
uneven number of keys:
https://github.com/sqlite/sqlite/blob/master/src/vdbeaux.c#L4719

Reviewed-by: Preston Thorpe (@PThorpe92)

Closes #1215
2025-03-31 10:40:01 +03:00
Pekka Enberg
5a60815484 core/json: Remove redundant conditional compilation
The top-level `json` module declaration is already conditional on "json"
feature so no need to do it here.
2025-03-31 09:40:01 +03:00
Pekka Enberg
6258dda5f1 core/json: Rename json_path.rs to path.rs 2025-03-31 09:40:01 +03:00
Pekka Enberg
4a91988755 core/json: Rename json_operations.rs to ops.rs 2025-03-31 09:33:30 +03:00
Pekka Enberg
4b77f52bac core/json: Rename json_cache.rs to cache.rs 2025-03-31 09:32:34 +03:00
Pekka Enberg
b8760c1cb0 Merge 'Jsonb support update' from Ihor Andrianov
1. Removed dependecy on serde, now all json functions use jsonb
implementation.
2. Add support for jsonb_patch.
3. Aggregation json functions like json_group_array, json_group_object.

Closes #1207
2025-03-31 09:29:28 +03:00
Jussi Saurio
42e25d23dd Fix IdxCmp insn comparisons 2025-03-30 23:01:41 +03:00
PThorpe92
211c9a0212 Remove From impl on iteration direction for sort order 2025-03-30 12:18:12 -04:00
PThorpe92
516e443a2b Fix use index cursor id in emitter and prevent reinsert pk on update 2025-03-30 12:15:25 -04:00
PThorpe92
43d3900777 Remove update limit tests from compat tcl tests 2025-03-30 12:15:25 -04:00
PThorpe92
7b25895940 Add update query tests with limit 2025-03-30 12:15:25 -04:00
PThorpe92
ff02d74afb Fix update queries when limit is 0 2025-03-30 12:15:25 -04:00
PThorpe92
a88ce2a4b7 Correct comment on update plan 2025-03-30 12:15:25 -04:00
PThorpe92
3f4636196c Fix error message for non btree table 2025-03-30 12:15:25 -04:00
PThorpe92
bbbd1df1ab Replace unwrap in update translation with parse error 2025-03-30 12:15:24 -04:00
PThorpe92
4ac0781991 Add update tcl tests for LIMIT clauses on update queries 2025-03-30 12:15:24 -04:00