Commit Graph

530 Commits

Author SHA1 Message Date
Pekka Enberg
e406a030e6 Merge 'Rework io_uring feature' from Jorge López Tello
This makes io_uring the default in CLI, but makes it non-default in
core. Before, if one built CLI without io_uring, core still built with
it as it was a default feature. To accommodate for the change, all
bindings have been updated to select the feature, except for WASM which
has a separate fs implementation.
This also adds some #[cfg] and #[allow] to silence unused-* warnings,
which I discovered when testing with different features disabled.

Closes #942
2025-02-09 08:46:29 +02:00
Jorge López
83b158fb3a core: silence some unused warnings when building without default features 2025-02-09 01:13:12 +01:00
Jussi Saurio
781aa3b5d6 sqlite3-parser: box the having clause in GroupBy 2025-02-08 18:10:26 +02:00
Jussi Saurio
670dac5939 sqlite3-parser: box the where clause in Delete 2025-02-08 18:10:25 +02:00
Pekka Enberg
8c0c967ea2 Merge 'Implement json_quote' from Pedro Muniz
Hi! This is my first PR on the project, so I apologize if I did not
follow a convention from the project.
#127
This PR implements json_quote as specified in their source:
https://www.sqlite.org/json1.html#jquote. It follows the internal doc
guidelines for implementing functions. Most tests were added from sqlite
test suite for json_quote, while some others were added by me. Sqlite
test suite for json_quote depends on json_valid to test for correct
escape control characters, so that specific test at the moment cannot be
done the same way.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Sonny (@sonhmai)

Closes #763
2025-02-07 13:33:05 +02:00
pedrocarlo
26388cc802 fix: cargo fmt 2025-02-06 23:36:02 -03:00
pedrocarlo
303a687e65 rebase to main 2025-02-06 23:35:58 -03:00
Jussi Saurio
d5f58f5fea Add quickcheck tests for generate_series() and refine implementation 2025-02-06 18:36:21 +02:00
PThorpe92
ae88d51e6f Remove TableReferenceType enum to clean up planner 2025-02-06 09:15:39 -05:00
PThorpe92
d4c06545e1 Refactor vtable impl and remove Rc Refcell from module 2025-02-06 09:15:39 -05:00
Jussi Saurio
f5f77c0bd1 Initial virtual table implementation 2025-02-06 07:51:50 -05:00
Pekka Enberg
f3902ef9b6 core: Rename OwnedRecord to Record
We only have one record type so let's call it `Record`.
2025-02-06 13:40:34 +02:00
krishvishal
32080aba5d Make vector function accessible through Function op code. 2025-02-06 07:01:50 +05:30
Pekka Enberg
6ea7fa06d2 Merge 'prepare perf: make ProgramBuilder aware of plan to count/estimate required memory' from Jussi Saurio
Use knowledge of query plan to inform how much memory to initially
allocate for `ProgramBuilder` vectors
Some of them are exact, some are semi-random estimates
```sql
Prepare `SELECT 1`/Limbo/SELECT 1
                        time:   [756.93 ns 758.11 ns 759.59 ns]
                        change: [-4.5974% -4.3153% -4.0393%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low severe
  1 (1.00%) low mild
  3 (3.00%) high mild
  1 (1.00%) high severe

Prepare `SELECT * FROM users LIMIT 1`/Limbo/SELECT * FROM users LIMIT 1
                        time:   [1.4739 µs 1.4769 µs 1.4800 µs]
                        change: [-7.9364% -7.7171% -7.4979%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou...`
                        time:   [3.7440 µs 3.7520 µs 3.7596 µs]
                        change: [-5.4627% -5.1578% -4.8445%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
```

Closes #899
2025-02-05 18:24:16 +02:00
Pekka Enberg
b5f5e40986 Merge 'prepare perf: dont eagerly allocate result column name strings' from Jussi Saurio
- Remove eagerly allocated `name` from `ResultSetColumn`
- `ResultSetColumn` can calculate `name()` on demand:
    - if it has an alias (`foo as bar`), use that
    - if it is a column reference, use that
    - otherwise return none, and callers can assign it a placeholder
name (like `column_1`)
- move the `plan.result_columns` and `plan.table_references` to
`Program` after preparing statement is done, so that column names can be
returned upon request
- make `name` in `Column` optional, not needed for pseudo tables and
sorters so avoids an extra string allocation
```sql
Prepare `SELECT 1`/Limbo/SELECT 1
                        time:   [756.80 ns 758.27 ns 760.04 ns]
                        change: [-3.3257% -3.0252% -2.7035%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
  1 (1.00%) high mild
  2 (2.00%) high severe

Prepare `SELECT * FROM users LIMIT 1`/Limbo/SELECT * FROM users LIMIT 1
                        time:   [1.4646 µs 1.4669 µs 1.4696 µs]
                        change: [-6.4769% -6.2021% -5.9137%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low severe
  3 (3.00%) low mild
  3 (3.00%) high severe

Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou...`
                        time:   [3.7256 µs 3.7311 µs 3.7376 µs]
                        change: [-4.5195% -4.2192% -3.9309%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  2 (2.00%) high mild
```

Closes #898
2025-02-05 18:20:01 +02:00
Jussi Saurio
795576b2ec dont eagerly allocate result column name strings 2025-02-05 17:53:23 +02:00
Jussi Saurio
f599b5a752 Make programbuilder aware of plan to count/estimate required memory 2025-02-05 14:22:42 +02:00
Pekka Enberg
56d401fb67 Merge 'Implement json_set' from Marcus Nilsson
This PR adds support for `json_set`.
There are three helper functions added:
1. `json_path_from_owned_value`, this function turns an `OwnedValue`
into a `JsonPath`.
2. `find_or_create_target`, this function is similar to `find_target`
with the added bonus of creating the target if it doesn't exist. There
is a caveat with this function and that is that it will create
objects/arrays as it goes, meaning if you send `{}` into it and try
getting the path `$.some.nested.array[123].field`, it will return
`{"some":{"nested":array:[]}}` since creation of `some`, `nested` and
`array` will succeed, but accessing element `123` will fail.
3. `create_and_mutate_json_by_path`, this function is very similar to
`mutate_json_by_path` but calls `find_or_create_target` instead of
`find_target`

Related to #127

Closes #878
2025-02-05 14:15:02 +02:00
Pekka Enberg
9fdf54de2b Merge 'Small perf optimizations to statement preparation' from Jussi Saurio
```bash
Prepare `SELECT 1`/Limbo/SELECT 1
                        time:   [765.94 ns 768.26 ns 771.03 ns]
                        change: [-7.8340% -7.4887% -7.1406%] (p = 0.00 < 0.05)
                        Performance has improved.

Prepare `SELECT * FROM users LIMIT 1`/Limbo/SELECT * FROM users LIMIT 1
                        time:   [1.5673 µs 1.5699 µs 1.5731 µs]
                        change: [-10.810% -9.7122% -8.4951%] (p = 0.00 < 0.05)
                        Performance has improved.

Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou...
                        time:   [4.1331 µs 4.1421 µs 4.1513 µs]
                        change: [-9.3157% -9.0255% -8.7372%] (p = 0.00 < 0.05)
                        Performance has improved.
```
flamegraph for prepare `SELECT 1`:
<img width="1718" alt="Screenshot 2025-02-03 at 10 34 14"
src="https://github.com/user-
attachments/assets/ba67fe2f-78b2-4796-9a09-837d8e79fe62" />

Closes #872
2025-02-05 10:46:57 +02:00
Marcus Nilsson
01492cf46f add support for json_set
Test cases are included.
Related to #127
2025-02-04 19:09:58 +01:00
Pekka Enberg
bf1ef13c91 Merge 'Add Printf Support' from Zaid Humayun
Add basic printf function support in limbo
![Screenshot 2025-02-04 at 8 08 23 PM](https://github.com/user-
attachments/assets/b12931eb-8e79-4c8a-af77-c25c34cc5834)

Closes #886
2025-02-04 17:53:27 +02:00
Zaid Humayun
6a863b3da9 printf: this commit adds support for https://github.com/tursodatabase/limbo/issues/885 tracking printf functionality
this commit introduces basic support for printf functionality and doesn't include advanced modifiers like width etc.
2025-02-04 20:05:14 +05:30
Pekka Enberg
2d9a54ccba Merge 'implement json_pretty' from Pedro Muniz
This PR implements json_pretty. At the moment, support for jsonb is
being added, so this function suffers from the same limitations as in
json(x). Also, I have not found a way to implement the same conversion
of Blob -> String that SQLite does. From my own experimentation, I
believe SQLite converts blobs to a lossy ascii representation, but I
would appreciate some help on this.

Closes #860
2025-02-04 14:51:47 +02:00
Pekka Enberg
0050f4aeea Merge 'Support column aliases in GROUP BY, ORDER BY and HAVING' from Jussi Saurio
Closes #744
```sql
# Wanda = 9, Whitney = 11, William = 111
do_execsql_test column_alias_in_group_by_order_by_having {
  select first_name as fn, count(1) as fn_count from users where fn in ('Wanda', 'Whitney', 'William') group by fn having fn_count > 10 order by fn_count;
} {Whitney|11
William|111}
```

Closes #864
2025-02-04 14:51:05 +02:00
Jussi Saurio
d182ddf514 dont store insn comments unless the query is EXPLAIN 2025-02-03 19:53:33 +02:00
Jussi Saurio
750a9c6463 assertions and small cleanups 2025-02-03 13:08:13 +02:00
Jussi Saurio
40f536fabb Dont store available_indexes on plan; only used in optimize_plan() 2025-02-03 12:52:14 +02:00
Jussi Saurio
61a007fb29 preallocate plan.result_columns according to AST 2025-02-03 12:51:24 +02:00
Jussi Saurio
f2dab8499d preallocate loop metadata according to table/column count and prefer vec over hashmap 2025-02-03 12:51:24 +02:00
Jussi Saurio
d4cb0a1223 Merge 'Fix logical codegen' from Nikita Sivukhin
Fix few logical codegen issues and add fuzz tests for logical
expressions
-  Right now Limbo fails to recognize `false` constant in case when any
unary operator is used on the AST path. This PR add unary operator
option in the rewrite code and handle such cases properly.
```sql
limbo> SELECT NOT FALSE;

  × Parse error: no such column: FALSE - should this be a string literal in single-quotes?

```
- `ifnull` implementation produced incorrect codegen due to "careless"
management of registers
```
limbo> SELECT ifnull(0, NOT 0)
[NULL here]
```
- `like` implementation produced incorrect codegen due to "careless"
management of registers
```
limbo> SELECT like('a%', 'a') = 1;
thread 'main' panicked at core/vdbe/mod.rs:1902:41:
internal error: entered unreachable code: Like on non-text registers
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Depends on https://github.com/tursodatabase/limbo/pull/867 (need
`GrammarGenerator` from this branch)

Closes #869
2025-02-03 12:39:41 +02:00
Jussi Saurio
1e5501650a Support column aliases in GROUP BY, ORDER BY and HAVING 2025-02-03 10:44:05 +02:00
Nikita Sivukhin
979612cb34 fix miscompilation of like function 2025-02-03 11:25:14 +04:00
Nikita Sivukhin
11c47f5e44 fix miscomplation of ifnull scalar function 2025-02-03 11:25:14 +04:00
Nikita Sivukhin
a4a80f37bc rewrite unary expressions too - in order to support "NOT FALSE" expressions 2025-02-03 11:25:14 +04:00
Nikita Sivukhin
5a3587f7a2 use opposite operator for search if WHERE condition is swapped (e.g. 1 > x instead of x < 1) 2025-02-03 11:23:04 +04:00
Pekka Enberg
7257fb8aae Merge 'core: move pragma statement bytecode generator to its own file.' from Sonny
What?
- no logic change
- refactored and moved pragma statement bytecode generation to its own
package to better structure.

Closes #871
2025-02-03 09:10:33 +02:00
Pekka Enberg
662d629666 Rename JoinAwareConditionExpr to WhereTerm
We transform all JOIN conditions into WHERE clause terms in the query
planner. The JoinAwareConditionExpr name tries to make that point, but I
think it makes things more confusing. Let's call it WhereTerm (suggested
by Jussi).
2025-02-03 07:46:51 +02:00
sonhmai
2d4bf2eb62 core: move pragma statement bytecode generator to its own file. 2025-02-03 09:21:14 +07:00
Jussi Saurio
c18c6ad64d Marginal changes to use new data structures and field names 2025-02-02 10:18:13 +02:00
Jussi Saurio
82a2850de9 subquery.rs: use iteration instead of recursion and simplify 2025-02-02 10:18:13 +02:00
Jussi Saurio
98439cd936 optimizer.rs: refactor to use new data structures and remove unnecessary stuff
We don't need `push_predicates()` because that never REALLY was a predicate
pushdown optimization -- it just pushed WHERE clause condition expressions
into the correct SourceOperator nodes in the tree.

Now that we don't have a SourceOperator tree anymore and we keep the conditions
in the WHERE clause instead, we don't need to "push" anything anymore. Leaves
room for ACTUAL predicate pushdown optimizations later :)

We also don't need any weird bitmask stuff anymore, and perhaps we never did,
to determine where conditions should be evaluated.
2025-02-02 10:18:13 +02:00
Jussi Saurio
89fba9305a main_loop.rs: use iteration instead of recursion
Now that we do not have a tree of SourceOperators but rather
a Vec of TableReferences, we can just use loops instead of
recursion for handling the main query loop.
2025-02-02 10:18:13 +02:00
Jussi Saurio
09b6bad0af delete.rs: use new data structures when parsing delete 2025-02-02 10:18:13 +02:00
Jussi Saurio
2ddac4bf21 select.rs: use new data structures when parsing select 2025-02-02 10:18:13 +02:00
Jussi Saurio
16a97d3b98 planner.rs: refactor from/join + where parsing logic
- use new TableReference and JoinAwareConditionExpr
- add utilities for determining at which loop depth a
  WHERE condition should be evaluated, now that "operators"
  do not carry condition expressions inside them anymore.
2025-02-02 10:18:13 +02:00
Jussi Saurio
e63256f657 Change Display implementation of Plan to work with new data structures 2025-02-02 10:18:13 +02:00
Jussi Saurio
390d0e673f plan.rs: refactor data structures
- Get rid of SourceOperator tree
- Make plan have a Vec of TableReference, and TableReference now
  contains the information from the old SourceOperator.
- Remove `predicates` (conditions) from Table References -- put
  everything in the WHERE clause like SQLite, and attach metadata
  to the where clause expressions with JoinAwareConditionExpr struct.
- Refactor select_star() to be simpler now that we use a vec, not a tree
2025-02-02 10:18:13 +02:00
Pekka Enberg
dbb7d1a6ba Merge 'Pagecount' from Glauber Costa
This PR implements the Pagecount pragma, as well as its associated
bytecode opcode

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #819
2025-02-02 09:32:18 +02:00
pedrocarlo
2e115d948d implement json_pretty 2025-02-01 23:04:46 -03:00
Glauber Costa
a3387cfd5f implement the pragma page_count
To do that, we also have to implement the vdbe opcode Pagecount.
2025-02-01 19:39:46 -05:00