Commit Graph

3695 Commits

Author SHA1 Message Date
PThorpe92
22fd3e9781 Fix cli tests 2025-04-03 15:23:49 -04:00
Pekka Enberg
73a35329d0 Merge 'Fix overflow position in write_page()' from Lâm Hoàng Phúc
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1241
2025-04-03 16:33:20 +03:00
Pekka Enberg
ccbea57a4d Merge 'docs(readme): update discord link' from Jamie Barton
Closes #1242

Closes #1243
2025-04-03 15:35:16 +03:00
Jamie Barton
83ec159b0c docs(readme): update discord link 2025-04-03 13:09:24 +01:00
Pekka Enberg
67627e18c8 Limbo 0.0.19-pre.4 2025-04-03 13:36:01 +03:00
Pekka Enberg
4392193856 bindings/javascript: Fix package repository URLs 2025-04-03 13:34:30 +03:00
Pekka Enberg
4f34373392 Limbo 0.0.19-pre.3 2025-04-03 12:39:17 +03:00
Pekka Enberg
64bd0c141a github: Fix JavaScript publish workflow permissions
The workflow enables npm's provenance feature, which requires write
permissions to the "id-token" on Github actions:

https://tsmx.net/npmjs-built-and-signed-on-github-actions/
2025-04-03 12:38:01 +03:00
TcMits
56fa9049c3 fix: overflow pos in write_page 2025-04-03 15:02:53 +07:00
Pekka Enberg
a5ee6493c0 Limbo 0.0.19-pre.2 2025-04-03 10:43:36 +03:00
Pekka Enberg
4342438801 scripts: Add more npm packages to update-version.py 2025-04-03 10:43:28 +03:00
Pekka Enberg
a279056e88 bindings/javascript: Drop pre-publish step
We don't want napi creating releases.
2025-04-03 10:41:15 +03:00
Pekka Enberg
7075c75b24 Limbo 0.0.19-pre.1 2025-04-03 10:03:01 +03:00
Pekka Enberg
c61063c697 github: Fix JavaScript workflow release parsing 2025-04-03 10:03:01 +03:00
Pekka Enberg
16bc28b0af sqlite3-parser: Change debug logging to trace level
SQL scanner at debug level spams the logs pretty hard when debugging...
2025-04-03 07:46:07 +03:00
Pekka Enberg
7b9c0e9231 Merge 'More VDBE cleanups' from Pekka Enberg
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1237
2025-04-02 19:25:41 +03:00
Pekka Enberg
24063bd9c0 core/vdbe: Add newline between op functions 2025-04-02 18:57:07 +03:00
Pekka Enberg
c592e27dca core/vdbe: Move explain() to last method in Program
...it's the least interesting one and we should have `step()` at the top.
2025-04-02 18:55:35 +03:00
Pekka Enberg
ed1854c8de Merge 'Request load page on insert_into_page' from Pere Diaz Bou
We assumed page was loaded because before inserting we would move there.
`NewRowId` unfortunately moves cursor to the rightmost page causing
eviction of root page -- this arose the issue with `insert_into_page`
not loading the page we were supposed to have loaded so I added
`return_if_locked_maybe_load` which is a utility macro to check if the
page is locked and if not, load it if needed.

Closes #1138
2025-04-02 18:52:25 +03:00
Pere Diaz Bou
65c4cb1e0e Merge 'core/vdbe: Rename execute_insn_* to op_*' from Pekka Enberg
The "execute::execute_insn" prefix is noisy, let's rename the
instruction operation functions to something shorter and sweeter.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1235
2025-04-02 17:25:12 +02:00
Pekka Enberg
265457f175 Merge 'Don't emit Transaction for simple SELECT statements' from Diego Reis
First step to close #1226.
Before:
![image](https://github.com/user-
attachments/assets/990bc6e2-a8f1-44db-9f82-f70a430663c1)
After:
![image](https://github.com/user-
attachments/assets/a587e207-d10f-487b-97b1-47b46f5dceff)
Reusing the same register is a bit trickier, I'm understanding how
SQLite does this optimization to apply here as well.
EDIT: Now we reuse the register and have the same number of bytecodes as
SQLite.
![image](https://github.com/user-
attachments/assets/bd7d769b-a680-4e77-ac2d-a3f1728bfdb7)

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1227
2025-04-02 18:12:20 +03:00
Pekka Enberg
3420955db7 core/vdbe: Rename execute_insn_* to op_*
The "execute::execute_insn" prefix is noisy, let's rename the
instruction operation functions to something shorter and sweeter.
2025-04-02 18:02:02 +03:00
Pere Diaz Bou
5dedc68fda remove arc import 2025-04-02 16:56:34 +02:00
Pekka Enberg
ee203e30ba Merge 'Remove RWLock from Shared wal state' from Pere Diaz Bou
Fixes #780
WalShared state can be shared without having to wrap everything with a
lock, and instead use atomics on some places and rwlock on others -- for
now.
## Results:
From:
----
```
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [34.125 ns 34.218 ns 34.324 ns]
Execute `SELECT 1`/sqlite_execute_select_1
                        time:   [28.124 ns 28.254 ns 28.385 ns]
```
To:
----
```bash
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [31.919 ns 32.113 ns 32.327 ns]
Execute `SELECT 1`/sqlite_execute_select_1
                        time:   [29.662 ns 29.900 ns 30.139 ns]

```
And with `begin_read_tx` inlined:
----
```bash
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [30.543 ns 30.585 ns 30.632 ns]
```

Closes #1225
2025-04-02 17:47:45 +03:00
Pere Diaz Bou
e85fb86ff4 Request load page on insert_into_page
We assumed page was loaded because before inserting we would move there. `NewRowId` unfortunately moves cursor to the rightmost page causing eviction of root page -- this arose the issue with `insert_into_page` not loading the page we were supposed to have loaded so I added `return_if_locked_maybe_load` which is a utility macro to check if the page is locked and if not, load it if needed.
2025-04-02 16:24:53 +02:00
Pere Diaz Bou
46814d2bd7 ignore warning mut_from_ref 2025-04-02 16:18:36 +02:00
Pere Diaz Bou
e2d00a9f96 inline start transactions from pager and wal
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [30.543 ns 30.585 ns 30.632 ns]
2025-04-02 16:18:36 +02:00
Pere Diaz Bou
2a49fe9bd2 Remove RWLock from Shared wal state
WalShared state can be shared without having to wrap everything with a
lock, and instead use atomics on some places and rwlock on others -- for
now.

## Results:
From:
----
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [34.125 ns 34.218 ns 34.324 ns]
Execute `SELECT 1`/sqlite_execute_select_1
                        time:   [28.124 ns 28.254 ns 28.385 ns]

To:
----
Gnuplot not found, using plotters backend
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [31.919 ns 32.113 ns 32.327 ns]
Execute `SELECT 1`/sqlite_execute_select_1
                        time:   [29.662 ns 29.900 ns 30.139 ns]
2025-04-02 16:18:36 +02:00
Pekka Enberg
cd5ef7c7db Merge 'Reuse register in binary expressions if they're equal ' from Diego Reis
Alongside with #1227, this PR closes #1226
Before:
![image](https://github.com/user-attachments/assets/d9fd4ff4-957d-46cd-
abe0-63d0b2c4acdb)
After:
![image](https://github.com/user-
attachments/assets/2ce4b8f1-f8f8-423a-8d35-d947de2794cb)

Closes #1234
2025-04-02 17:02:35 +03:00
Pekka Enberg
0d3e92d42f Merge 'VDBE with indirect function dispatch' from Pere Diaz Bou
This PR is unapologetically stolen from @vmg's implementation in Vitess
implemented here https://github.com/vitessio/vitess/pull/12369. If you
want a more in depth explanation of how this works you can read the
[blog post he carefully
wrote](https://planetscale.com/blog/faster-interpreters-in-go-catching-
up-with-cpp).
In limbo we have a huge problem with [register
spilling](https://en.wikipedia.org/wiki/Register_allocation), this can
be easily observed with the prolog of `Program::step` before:
```llvm
start:
    %e.i.i304.i = alloca [0 x i8], align 8
    %formatter.i305.i = alloca [64 x i8], align 8
    %buf.i306.i = alloca [24 x i8], align 8
    %formatter.i259.i = alloca [64 x i8], align 8
    ..................... these are repeated for hundreds of lines
.....................
    %formatter.i52.i = alloca [64 x i8], align 8
    %buf.i53.i = alloca [24 x i8], align 8
    %formatter.i.i = alloca [64 x i8], align 8
    %buf.i.i = alloca [24 x i8], align 8
    %_87.i = alloca [48 x i8], align 8
    %_82.i = alloca [24 x i8], align 8
    %_73.i = alloca [24 x i8], align 8
    %_66.i8446 = alloca [24 x i8], align 8
    %_57.i = alloca [24 x i8], align 8
    %_48.i = alloca [24 x i8], align 8
```
After these changes we completely remove the need of register spilling
(yes that is the complete prolog):
```llvm
start:
    %self1 = alloca [80 x i8], align 8
    %pager = alloca [8 x i8], align 8
    %mv_store = alloca [8 x i8], align 8
    store ptr %0, ptr %mv_store, align 8
    store ptr %1, ptr %pager, align 8
    %2 = getelementptr inbounds i8, ptr %state, i64 580
    %3 = getelementptr inbounds i8, ptr %state, i64 576
    %4 = getelementptr inbounds i8, ptr %self, i64 16
    %5 = getelementptr inbounds i8, ptr %self, i64 8
    %6 = getelementptr inbounds i8, ptr %self1, i64 8
    br label %bb1, !dbg !286780
```
When it comes to branch prediction, we don't really fix a lot because
thankfully rust already compiles `match` expressions to a jump table:
```llvm
%insn = getelementptr inbounds [0 x %"vdbe::insn::Insn"], ptr %self657,
i64 0, i64 %index, !dbg !249527
%332 = load i8, ptr %insn, align 8, !dbg !249528, !range !38291,
!noundef !14
switch i8 %332, label %default.unreachable26674 [
    i8 0, label %bb111
    i8 1, label %bb101
    i8 2, label %bb100
    i8 3, label %bb110
    ...
    i8 104, label %bb5
    i8 105, label %bb16
    i8 106, label %bb14
], !dbg !249530
```
Some results
----
```
function dispatch:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [29.498 ns 29.548 ns 29.601 ns]
                        change: [-3.6125% -3.3592% -3.0804%] (p = 0.00 <
0.05)

main:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [33.789 ns 33.832 ns 33.878 ns]
```

Closes #1233
2025-04-02 17:01:58 +03:00
Pere Diaz Bou
66f70d571d fmt 2025-04-02 13:14:26 +00:00
Pere Diaz Bou
f5221589f0 remove wrong usage of feature = json 2025-04-02 15:00:51 +02:00
Pere Diaz Bou
7e4b57f2e2 VDBE with direct function dispatch
This PR is unapologetically stolen from @vmg's implementation in Vitess
implemented here https://github.com/vitessio/vitess/pull/12369. If you
want a more in depth explanation of how this works you can read the
[blog post he carefully
wrote](https://planetscale.com/blog/faster-interpreters-in-go-catching-up-with-cpp).

In limbo we have a huge problem with [register
spilling](https://en.wikipedia.org/wiki/Register_allocation), this can
be easily observed with the prolog of `Program::step` before:
```llvm
start:
    %e.i.i304.i = alloca [0 x i8], align 8
    %formatter.i305.i = alloca [64 x i8], align 8
    %buf.i306.i = alloca [24 x i8], align 8
    %formatter.i259.i = alloca [64 x i8], align 8
    ..................... these are repeated for hundreds of lines
.....................
    %formatter.i52.i = alloca [64 x i8], align 8
    %buf.i53.i = alloca [24 x i8], align 8
    %formatter.i.i = alloca [64 x i8], align 8
    %buf.i.i = alloca [24 x i8], align 8
    %_87.i = alloca [48 x i8], align 8
    %_82.i = alloca [24 x i8], align 8
    %_73.i = alloca [24 x i8], align 8
    %_66.i8446 = alloca [24 x i8], align 8
    %_57.i = alloca [24 x i8], align 8
    %_48.i = alloca [24 x i8], align 8
```

After these changes we completely remove the need of register spilling
(yes that is the complete prolog):
```llvm
start:
    %self1 = alloca [80 x i8], align 8
    %pager = alloca [8 x i8], align 8
    %mv_store = alloca [8 x i8], align 8
    store ptr %0, ptr %mv_store, align 8
    store ptr %1, ptr %pager, align 8
    %2 = getelementptr inbounds i8, ptr %state, i64 580
    %3 = getelementptr inbounds i8, ptr %state, i64 576
    %4 = getelementptr inbounds i8, ptr %self, i64 16
    %5 = getelementptr inbounds i8, ptr %self, i64 8
    %6 = getelementptr inbounds i8, ptr %self1, i64 8
    br label %bb1, !dbg !286780
```
When it comes to branch prediction, we don't really fix a lot because
thankfully rust already compiles `match` expressions
to a jump table:

```llvm
%insn = getelementptr inbounds [0 x %"vdbe::insn::Insn"], ptr %self657,
i64 0, i64 %index, !dbg !249527
%332 = load i8, ptr %insn, align 8, !dbg !249528, !range !38291,
!noundef !14
switch i8 %332, label %default.unreachable26674 [
    i8 0, label %bb111
    i8 1, label %bb101
    i8 2, label %bb100
    i8 3, label %bb110
    ...
    i8 104, label %bb5
    i8 105, label %bb16
    i8 106, label %bb14
], !dbg !249530
```

Some results
----
```
function dispatch:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [29.498 ns 29.548 ns 29.601 ns]
                        change: [-3.6125% -3.3592% -3.0804%] (p = 0.00 <
0.05)

main:
Execute `SELECT 1`/limbo_execute_select_1
                        time:   [33.789 ns 33.832 ns 33.878 ns]
```
2025-04-02 14:55:37 +02:00
Diego Reis
3c531ac5ec core/expr: Reuse register in binary expressions if they're equal 2025-04-02 09:15:41 -03:00
Diego Reis
86f8719b69 core/expr: Extract binary insn emission in a separate function 2025-04-02 09:14:01 -03:00
Pekka Enberg
65ae698773 Limbo 0.0.18 2025-04-02 15:04:48 +03:00
Pekka Enberg
f7d76e0fab Update CHANGELOG.md 2025-04-02 15:04:39 +03:00
Diego Reis
fe37046536 core/emitter: Don't emit for statements that doesn't reference any tables 2025-04-02 08:59:52 -03:00
Diego Reis
6660a99003 core/emitter: Refactor the epilogue function to receive an enum 2025-04-02 08:56:19 -03:00
Pekka Enberg
e79da7375b Limbo 0.0.18-pre.5 2025-04-02 13:38:22 +03:00
Pekka Enberg
7394ad6854 Disable more b-tree fuzzers... 2025-04-02 13:38:09 +03:00
Pekka Enberg
d9562a3d82 Add update-version.py script
Simplifies version bumping.
2025-04-02 09:31:28 +03:00
Pekka Enberg
f74a10c9c1 Limbo 0.0.18-pre.4 2025-04-02 09:30:42 +03:00
Pekka Enberg
cc8340d30e Disable btree_insert_fuzz_run_random test 2025-04-02 09:15:01 +03:00
Pekka Enberg
6199c3994a Merge 'Create plan for Update queries' from Preston Thorpe
closes #1186, or at least works towards it by implementing an actual
Plan for update queries instead of translating everything inline. This
allows for actually using index's, rewriting const expressions, pushing
predicates instead of hardcoding a full scan in the translation.
### TODOs:
1.  `RETURNING` clause/result columns
2.  `OFFSET` clauses
3. on conflict
### LIMIT:
By supporting `LIMIT` directly in update queries, we'll have to put the
tests outside of the compatibility tests, maybe in the CLI tests.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1189
2025-04-01 17:33:32 +03:00
Pekka Enberg
149f4aa729 Merge 'Remove unnecessary balance code that crashes' from Pere Diaz Bou
Closes #1223
2025-04-01 17:31:23 +03:00
Pekka Enberg
33b123bc2a Merge 'Validate cells inside a page after each operation' from Pere Diaz Bou
We need to ensure an operation doesn't transform the cells inside a page
to an invalid state. In debug mode we can enable a
`debug_validate_cells` with `#[cfg(debug_assertions)]` so that it is
skipped on release mode. This is good to catch things early instead of
seeing the page in an incredibly weird state some state in the future.

Closes #1222
2025-04-01 17:31:08 +03:00
Pere Diaz Bou
6b05dbddb0 remove unnecessary code while building count old and size old balancing 2025-04-01 13:11:55 +02:00
Pere Diaz Bou
70c5cf3970 Merge 'Refactor Cli Repl Commands to use clap' from Pedro Muniz
This PR changes the argument parsing and matching to use Clap instead of
our own handrolled one. This makes it much easier to add new commands
and modify existing ones by using the full power of clap's derive
macros. It also produces nice error and help messages for us. However,
there is a bug in Clap that is not correctly modifying the `display
name` in the help section. So the command would appear as `show` instead
of `.show` in the help messages. This is very minimal, but if this is a
blocker for this PR we can just overwrite the help message in its
entirety. Also using Clap would enable us to use its autocomplete crate
to generate autocompletions for these special repl commands which would
be a huge win when compared to the `sqlite3` cli.
This is the current help message:
```sh

Limbo SQL Shell Help
==============
Welcome to the Limbo SQL Shell! You can execute any standard SQL command here.
In addition to standard SQL commands, the following special commands are available:

Usage: <COMMAND>

Commands:
  exit       Exit this program with return-code CODE
  quit       Quit the shell
  open       Open a database file
  schema     Print this message or the help of the given subcommand(s) Display schema for a table
  output     Set output file (or stdout if empty)
  mode       Set output display mode
  opcodes    Show vdbe opcodes
  cd         Change the current working directory
  show       Display information about settings
  nullvalue  Set the value of NULL to be printed in 'list' mode
  echo       Toggle 'echo' mode to repeat commands before execution
  tables     Display tables
  import     Import data from FILE into TABLE
  load       Loads an extension library
  dump       Dump the current database as a list of SQL statements
  listvfs    List vfs modules available
  help       Print this message or the help of the given subcommand(s)

Usage Examples:
---------------
1. To quit the Limbo SQL Shell:
   .quit

2. To open a database file at path './employees.db':
   .open employees.db

3. To view the schema of a table named 'employees':
   .schema employees

4. To list all tables:
   .tables

5. To list all available SQL opcodes:
   .opcodes

6. To change the current output mode to 'pretty':
   .mode pretty

7. Send output to STDOUT if no file is specified:
   .output

8. To change the current working directory to '/tmp':
   .cd /tmp

9. Show the current values of settings:
   .show

10. To import csv file 'sample.csv' into 'csv_table' table:
   .import --csv sample.csv csv_table

11. To display the database contents as SQL:
   .dump

12. To load an extension library:
   .load /target/debug/liblimbo_regexp

13. To list all available VFS:
   .listvfs

Note:
- All SQL commands must end with a semicolon (;).
- Special commands start with a dot (.) and are not required to end with a semicolon.

```
If we need more information on a specific command, we can leverage CLAP
and do for instance:
```sh
.open -h
```

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1110
2025-04-01 11:40:14 +02:00
Pere Diaz Bou
141303e330 Validate cells inside a page after each operation
We need to ensure an operation doesn't transform the cells inside a page
to an invalid state. In debug mode we can enable a
`debug_validate_cells` with `#[cfg(debug_assertions)]` so that it is
skipped on release mode.

Modify pager logs
2025-04-01 11:19:23 +02:00