Commit Graph

2687 Commits

Author SHA1 Message Date
Jussi Saurio
f388bc571e Merge 'xConnect for virtual tables to query core db connection' from Preston Thorpe
Re-Opening #1076 because it had bit-rotted to a point of no return.
However it has improved. Now with Weak references and no incrementing Rc
strong counts.
This also includes a better test extension that returns info about the
other tables in the schema.
![image](https://github.com/user-
attachments/assets/4292dc9c-121e-4ba2-8a51-4533bbcf2afd)
(theme doesn't show rows column)

Closes #1366
2025-05-25 14:37:38 +03:00
Jussi Saurio
621ae60ab5 Merge 'Reconstruct WAL frame cache when WAL is opened' from Jussi Saurio
Fixes #1567
Probably also fixes #1485
Currently we are simply unable to read any WAL frames from disk once a
fresh process w/ Limbo is opened, since we never try to read anything
from disk unless we already have it in our in-memory frame cache.
This commit implements a crude way of reading entire WAL into memory as
a single buffer and reconstructing the frame cache.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1570
2025-05-25 14:35:47 +03:00
Jussi Saurio
385c0d8987 clippy stfu part 2: electric boogaloo 2025-05-25 10:32:23 +03:00
Jussi Saurio
64ef3f1343 simplify condition 2025-05-25 10:22:46 +03:00
Jussi Saurio
20e65c0125 bump max_loops to 100k 2025-05-25 10:21:41 +03:00
Pere Diaz Bou
a91f8aee78 Merge 'set non-shared cache by default' from Pere Diaz Bou
Shared cache requires more locking mechasnisms. We still have multi
threading issues not related to shared cache so it is wise to first fix
those and then once they are fixed, we can incrementally add shared
cache back with locking in place.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1568
2025-05-25 08:50:21 +02:00
PThorpe92
98e1c0ddd4 Remove unused method for converting from ext type without ownership 2025-05-24 17:22:12 -04:00
PThorpe92
cf163f2dc0 Prevent double free in ext connection 2025-05-24 16:49:52 -04:00
PThorpe92
1cacbf1f0d Close statements in extension tests, and use mut pointers for stmt 2025-05-24 16:45:25 -04:00
PThorpe92
d63f9d8cff Make sure all resources are cleaned up properly in xconnect 2025-05-24 16:38:33 -04:00
PThorpe92
a4ed464ec4 Add some traces for errors in xconnect 2025-05-24 15:44:06 -04:00
PThorpe92
999205f896 Add documentation for connection api 2025-05-24 15:19:37 -04:00
Jussi Saurio
09df7e0c48 Merge 'TPC-H with criterion and nyrkio' from Pedro Muniz
Added a benchmark to bench TPC-H with criterion and have it uploaded to
Nyrkio. I did not delete the other tpc-h job because maybe someone uses
it and I'm not aware of it.

Closes #1560
2025-05-24 21:54:48 +03:00
PThorpe92
a2f8b2dfea Fix add check for invalid argv index for vtab constraints in main loop 2025-05-24 14:49:58 -04:00
PThorpe92
d11ef6b9c5 Add execute method to xConnect db interface for vtables 2025-05-24 14:49:58 -04:00
PThorpe92
c2ec6caae1 Finish integrating xConnect into vtable open api 2025-05-24 14:49:58 -04:00
PThorpe92
f61ccc78e8 Add from_ffi_ptr method to create OwnedValue from Ext type without taking ownership 2025-05-24 14:49:58 -04:00
PThorpe92
d51614a4fd Create extern functions to support vtab xConnect in core/ext 2025-05-24 14:49:57 -04:00
Jussi Saurio
fc45e0ec0d Reconstruct WAL frame cache when WAL is opened
Currently we are simply unable to read any WAL frames from disk
once a fresh process w/ Limbo is opened, since we never try to read
anything from disk unless we already have it in our in-memory
frame cache.

This commit implements a crude way of reading entire WAL into memory
as a single buffer and reconstructing the frame cache.
2025-05-24 18:29:44 +03:00
Jussi Saurio
8ed5334ca7 tests/fuzz: add compound_select_fuzz() 2025-05-24 13:12:41 +03:00
Jussi Saurio
f6443ae742 Support LIMIT with UNION ALL 2025-05-24 13:12:41 +03:00
Jussi Saurio
08bda9cc58 UNION ALL 2025-05-24 13:12:41 +03:00
Pere Diaz Bou
54b1647148 set non-shared cache by default
Shared cache requires more locking mechasnisms. We still have multi
threading issues not related to shared cache so it is wise to first fix
those and then once they are fixed, we can incrementally add shared
cache back with locking in place.
2025-05-24 11:59:54 +02:00
Jussi Saurio
0b2c3298aa Merge 'refactor: introduce walk_expr() and walk_expr_mut() to reduce repetitive pattern matching' from Jussi Saurio
We do a lot of
```rust
match expr {
  ...
}
```
just to find some specific case like `Expr::Column` deep inside an
expression tree. This PR introduces new helpers `walk_expr()` and
`walk_expr_mut()` that handle the tree-walking part, and the business
logic functions where this tree traversal is used can focus on the exact
enum variants of `Expr` that they are interested in.

Closes #1564
2025-05-23 22:00:20 +03:00
Jussi Saurio
70433e100d Merge 'btree: fix infinite looping in backwards iteration of btree table' from Jussi Saurio
Closes #1562
Existing "fuzz test" (not really fuzz, but kinda) didn't catch this due
to `LIMIT 3` clause

Closes #1563
2025-05-23 21:46:16 +03:00
pedrocarlo
1c4af7d0aa change sample count to 10 2025-05-23 12:37:03 -03:00
Jussi Saurio
2e095e6d03 Merge 'Add some comments for values statement' from meteorgan
follow up: #1549
simultaneously, address a warning in CI as the package `sqlite3-parser`
has been renamed to `limbo_sqlite3_parser`.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1565
2025-05-23 17:30:30 +03:00
meteorgan
3bf0ce7fb3 Add some comments for values statement 2025-05-23 22:11:34 +08:00
Jussi Saurio
1a937462b3 Merge 'core/pragma: Add support for update user_version' from Diego Reis
It also changes the type from u32 to i32 since
sqlite supports negative values

Closes #1559
2025-05-23 17:00:55 +03:00
Jussi Saurio
c18c6a00fa refactor: use walk_expr() in resolving vtab constraints 2025-05-23 16:28:56 +03:00
Jussi Saurio
fbfd2b2c38 refactor: use walk_expr_mut() in rewrite_expr() 2025-05-23 16:27:28 +03:00
Jussi Saurio
362347c474 refactor: use walk_expr() in determine_where_to_eval_expr() 2025-05-23 16:27:28 +03:00
Jussi Saurio
9ec84e3905 refactor: use walk_expr() in table_mask_from_expr() 2025-05-23 16:27:28 +03:00
Jussi Saurio
3835a29f47 refactor: use walk_expr() in resolve_aggregates() 2025-05-23 16:27:23 +03:00
Jussi Saurio
2ab5c5f6a9 refactor: use walk_expr_mut() in bind_column_references() 2025-05-23 15:56:49 +03:00
Jussi Saurio
40a4d162bc Introduce walker expressions for ast::Expr 2025-05-23 15:56:27 +03:00
Diego Reis
2d6405b3e9 core/pragma: Remove unnecessary clone in user_version and cache_size 2025-05-23 08:43:07 -03:00
Jussi Saurio
cbb56a182e Fix bug: backwards iteration of table btree hangs 2025-05-23 14:23:18 +03:00
Jussi Saurio
597020bc0c Merge 'Support values statement and values in select' from meteorgan
Close: #866
**limbo output**:
```
limbo> explain values(1, 2);
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     5     0                    0   Start at 5
1     Integer            1     1     0                    0   r[1]=1
2     Integer            2     2     0                    0   r[2]=2
3     ResultRow          1     2     0                    0   output=r[1..2]
4     Halt               0     0     0                    0
5     Goto               0     1     0                    0

limbo> explain values(1, 2), (3, 4);
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     16    0                    0   Start at 16
1     InitCoroutine      1     9     2                    0
2     Integer            1     2     0                    0   r[2]=1
3     Integer            2     3     0                    0   r[3]=2
4     Yield              1     0     0                    0
5     Integer            3     2     0                    0   r[2]=3
6     Integer            4     3     0                    0   r[3]=4
7     Yield              1     0     0                    0
8     EndCoroutine       1     0     0                    0
9     InitCoroutine      1     0     2                    0
10    Yield              1     15    0                    0
11    Copy               2     4     0                    0   r[4]=r[2]
12    Copy               3     5     0                    0   r[5]=r[3]
13    ResultRow          4     2     0                    0   output=r[4..5]
14    Goto               0     10    0                    0
15    Halt               0     0     0                    0
16    Goto               0     1     0                    0

limbo> explain select * from (values(1, 2), (3, 4));
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     16    0                    0   Start at 16
1     InitCoroutine      1     9     2                    0
2     Integer            1     2     0                    0   r[2]=1
3     Integer            2     3     0                    0   r[3]=2
4     Yield              1     0     0                    0
5     Integer            3     2     0                    0   r[2]=3
6     Integer            4     3     0                    0   r[3]=4
7     Yield              1     0     0                    0
8     EndCoroutine       1     0     0                    0
9     InitCoroutine      1     0     2                    0
10    Yield              1     15    0                    0
11    Copy               2     4     0                    0   r[4]=r[2]
12    Copy               3     5     0                    0   r[5]=r[3]
13    ResultRow          4     2     0                    0   output=r[4..5]
14    Goto               0     10    0                    0
15    Halt               0     0     0                    0
16    Transaction        0     0     0                    0   write=false
17    Goto               0     1     0                    0
```
**sqlite output**:
```
sqlite> explain values(1, 2);
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     5     0                    0   Start at 5
1     Integer        1     1     0                    0   r[1]=1
2     Integer        2     2     0                    0   r[2]=2
3     ResultRow      1     2     0                    0   output=r[1..2]
4     Halt           0     0     0                    0
5     Goto           0     1     0                    0
sqlite> explain values(1, 2), (3, 4);
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     16    0                    0   Start at 16
1     InitCoroutine  1     9     2                    0
2     Integer        1     4     0                    0   r[4]=1
3     Integer        2     5     0                    0   r[5]=2
4     Yield          1     0     0                    0
5     Integer        3     4     0                    0   r[4]=3
6     Integer        4     5     0                    0   r[5]=4
7     Yield          1     0     0                    0
8     EndCoroutine   1     0     0                    0
9     InitCoroutine  1     0     2                    0
10      Yield          1     15    0                    0   next row of 2-ROW VALUES CLAUSE
11      Copy           4     8     0                    2   r[8]=r[4]
12      Copy           5     9     0                    2   r[9]=r[5]
13      ResultRow      8     2     0                    0   output=r[8..9]
14    Goto           0     10    0                    0
15    Halt           0     0     0                    0
16    Goto           0     1     0                    0
sqlite>  explain select * from (values(1, 2), (3, 4));
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     16    0                    0   Start at 16
1     InitCoroutine  1     9     2                    0
2     Integer        1     4     0                    0   r[4]=1
3     Integer        2     5     0                    0   r[5]=2
4     Yield          1     0     0                    0
5     Integer        3     4     0                    0   r[4]=3
6     Integer        4     5     0                    0   r[5]=4
7     Yield          1     0     0                    0
8     EndCoroutine   1     0     0                    0
9     InitCoroutine  1     0     2                    0
10      Yield          1     15    0                    0   next row of 2-ROW VALUES CLAUSE
11      Copy           4     8     0                    2   r[8]=r[4]
12      Copy           5     9     0                    2   r[9]=r[5]
13      ResultRow      8     2     0                    0   output=r[8..9]
14    Goto           0     10    0                    0
15    Halt           0     0     0                    0
16    Goto           0     1     0                    0
```

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1549
2025-05-23 13:56:31 +03:00
meteorgan
01c8a4ca63 simpify values when it's subquery 2025-05-23 17:45:56 +08:00
Jussi Saurio
128a406f8c TableReference: fix stale comment 2025-05-23 10:06:24 +03:00
Zaid Humayun
bc5d93c18a Addresses comment https://github.com/tursodatabase/limbo/pull/1548#discussion_r2103333264 by @jussisaurio
this commit uses more descriptive names for registers
2025-05-23 10:19:08 +05:30
pedrocarlo
66f82e47f5 tpc-h with criterion and nyrkio 2025-05-23 01:32:16 -03:00
Diego Reis
2f8042da22 core/pragma: Add support for update user_version
It also changes the type from u32 to i32 since
sqlite supports negative values
2025-05-22 20:38:27 -03:00
Zaid Humayun
4312d371fb addresses comment https://github.com/tursodatabase/limbo/pull/1548#discussion_r2102606810 by @jussisaurio
this commit changes the btree_destroy() signature to return an Option<usize>. This more closely resembles Rust semantics instead of passing a pointer to a usize.
However, I'm unsure if I'm handling the cursor result correctly
2025-05-23 00:46:05 +05:30
Zaid Humayun
780d0cb5a8 addresses https://github.com/tursodatabase/limbo/pull/1548#discussion_r2102599487, https://github.com/tursodatabase/limbo/pull/1548#discussion_r2102601189 & https://github.com/tursodatabase/limbo/pull/1548#discussion_r2102602774 by @jussisaurio
this commit addresses comments regarding using decsriptive variable names for the loops and loop labels. Also, adds documentation for instructions that cause jumps in both loops
2025-05-23 00:31:19 +05:30
Zaid Humayun
3a8b18c481 addressed comment https://github.com/tursodatabase/limbo/pull/1548#discussion_r2102595246 by @jussisaurio
this commit addresses the comment regarding inline code documentation
2025-05-23 00:12:27 +05:30
Zaid Humayun
1999eac891 Fixes test Testing: can drop kv_store vtable
Earlier this test broke because the code to translate the drop table was not checking to see if a table was not a virtual table.
SQLite does this with a macro called IsVirtualTable. Here, I check with the existing methods on the BTree struct
2025-05-22 23:53:48 +05:30
meteorgan
34e05ef974 make values work in subquery 2025-05-23 00:30:04 +08:00
meteorgan
0467d7e11b Support values statement and values in select 2025-05-23 00:29:54 +08:00