I will warn that this PR is quite big out of necessity, since subqueries
are, as the name implies, queries within queries, so everything that
works with a regular query should also work with a subquery, roughly
speaking.
---
- Adds support for:
* uncorrelated subqueries in FROM clause (i.e. appear as a "table",
and do not refer to outer tables). Example of this at the end of the PR
description.
* column and subquery aliasing (`select sub.renamed from (select
name as renamed from products) sub`)
* inner and outer filtering of subqueries (`select sub.name from
(select name from products where name = 'joe') sub`, and, `select
sub.name from (select name from products) sub where sub.name = 'joe'`)
* joining between regular tables and subqueries
* joining between multiple subqueries
* in general working with subqueries should roughly equal working
with regular tables
- Main idea: subqueries are just wrappers of a `SelectPlan` that never
emit ResultRows, instead they `Yield` control back to the parent query,
and the parent query can copy the subquery result values into a
ResultRow. New variant `SourceOperator::Subquery` that wraps a subquery
`SelectPlan`.
- Plans can now not only refer to btree tables (`select p.name from
products`) but also subqueries (`select sub.foo from (select name as foo
from products) sub`. Hence this PR also adds support for column aliases
which didn't exist before.
* An `Expr::Column` that refers to a regular table will result in an
`Insn::Column` (i.e. a read from disk/memory) whereas an `Expr::Column`
that refers to a subquery will result in an `Insn::Copy` (from register
to register) instead
- Subquery handling is entirely unoptimized, there's no predicate
pushdown from outer query to subqueries, or elimination of redundant
subqueries (e.g. in the trivial example `SELECT * FROM (SELECT * FROM
users) sub` the subquery can just be entirely removed)
---
This PR does not add support (yet) for:
- subqueries in result columns: `SELECT t.foo, (SELECT .......) as
column_from_subquery FROM t`
- subqueries in WHERE clauses e.g. `SELECT * FROM t1 WHERE t1.foo IN
(SELECT ...)`
- subquery-related optimizations, of which there are plenty available.
No analysis is done regarding e.g. whether predicates on the outer query
level could be pushed into the subquery, or whether the subquery could
be entirely eliminated. Both of the above can probably be done fairly
easily for a bunch of trivial cases.
---
Example bytecode with comments added:
```
limbo> EXPLAIN SELECT p.name, sub.funny_name FROM products p JOIN (
select id, concat(name, '-lol') as funny_name from products
) sub USING (id) LIMIT 3;
addr opcode p1 p2 p3 p4 p5 comment
---- ----------------- ---- ---- ---- ------------- -- -------
0 Init 0 31 0 0 Start at 31
// Coroutine implementation starts at insn 2, jump immediately to 14
1 InitCoroutine 1 14 2 0
2 OpenReadAsync 0 3 0 0 table=products, root=3
3 OpenReadAwait 0 0 0 0
4 RewindAsync 0 0 0 0
5 RewindAwait 0 13 0 0 Rewind table products
6 RowId 0 2 0 0 r[2]=products.rowid
7 Column 0 1 4 0 r[4]=products.name
8 String8 0 5 0 -lol 0 r[5]='-lol'
9 Function 0 4 3 concat 0 r[3]=func(r[4..5])
// jump back to main loop of query (insn 20)
10 Yield 1 0 0 0
11 NextAsync 0 0 0 0
12 NextAwait 0 6 0 0
13 EndCoroutine 1 0 0 0
14 OpenReadAsync 1 3 0 0 table=p, root=3
15 OpenReadAwait 0 0 0 0
16 RewindAsync 1 0 0 0
17 RewindAwait 1 30 0 0 Rewind table p
// Since this subquery is the inner loop of the join, reinitialize it on every iteration of the outer loop
18 InitCoroutine 1 0 2 0
// Jump back to the subquery implementation to assign another row into registers
19 Yield 1 28 0 0
20 RowId 1 8 0 0 r[8]=p.rowid
// Copy sub.id
21 Copy 2 9 0 0 r[9]=r[2]
// p.id == sub.id?
22 Ne 8 9 27 0 if r[8]!=r[9] goto 27
23 Column 1 1 6 0 r[6]=p.name
// copy sub.funny_name
24 Copy 3 7 0 0 r[7]=r[3]
25 ResultRow 6 2 0 0 output=r[6..7]
26 DecrJumpZero 10 30 0 0 if (--r[10]==0) goto 30
27 Goto 0 19 0 0
28 NextAsync 1 0 0 0
29 NextAwait 1 18 0 0
30 Halt 0 0 0 0
31 Transaction 0 0 0 0
32 Integer 3 10 0 0 r[10]=3
33 Goto 0 1 0 0
```
Closes#566
The database object is a way to represent state that's shared across
multiple connections. We don't want to release that object until all
connections are closed.
EDIT: will continue iterating on these ideas, as discussed on discord.
for now, this has been changed to just enable `IORING_ENABLE_SQPOLL` by
default. This is supported sin `5.11`, and I believe the last debian
release < that reached EOL in July, so shouldn't be an issue.
Closes#557
Fixes#577
With the previous implementation we weren't escaping the regex meta
characters . And in certain cases glob had a different meaning than
regex.
For e.g , the below shows a glob pattern with its regex equivalent
- `[][]` translates to `[\]\[]`
- `[^][]` translates to `[^\]\[]`
Closes#578
I have added support for like function with escape i.e like(X,Y,Z) .
There is good opportunity to refactor/cleanup the like operations which
can be done in another PR, as I wanted to keep the changes small .
Closes#568
Fixes#552
In our construct regex function, we were not escaping the required
characters properly which was causing the failure.
Limbo output with this branch
```
limbo> select like('\%A', '\A');
1
limbo> select like('A$%', 'A$');
1
limbo> select like('%a.a', 'aaaa');
0
```
Closes#553
#560
Changes to `translate_expr` function:
* [`core/translate/expr.rs`](diffhunk://#diff-
371865d5d7b8bcaed649413c687492e61e94f21387cd9b2c47d989a033888c8bL1558-
R1560): Changed the `amount` parameter in the `Insn::Copy` instruction
from `1` to `0`.
Enhancements to the testing framework:
* [`testing/scalar-functions.test`](diffhunk://#diff-
a046d58ab24eee8207f0ce3199f8d0a609edcef9c24b8ed7f242f7a60e6c1e61R812-
R815): Added a new test `do_execsql_test_regex` to validate that the
`sqlite_version` function returns a valid output.
* [`testing/tester.tcl`](diffhunk://#diff-
316cca92d85df3f78558cc3e60d7420c1fd19a23ecf2bbea534db93ab08ea3ecR29-
R45): Introduced a new procedure `do_execsql_test_regex` to support
regex-based validation of SQL outputs.
Closes#561
The name "row result" is confusing because it really *is* a result from
a step() call. The only difference is how a row is represented as we
return from VDBE or from a statement.
Therefore, rename RowResult to StepResult.
In line with [other work](#127) for JSON support, this PR adds support
for [`json_array_length`](https://www.sqlite.org/json1.html#jarraylen).
This includes a first pass at supporting the JSON path for accessing
values within the JSON.
I've added tests in rust and tcl.
Closes#555
This PR's genesis is from investigating #532, but I still can't reliably
reproduce it on either `main` or this branch so I don't know if this PR
_fixes_ anything, but I guess it aligns us more with sqlite anyway
---
Anyway: I looked at DBs created with limbo and with sqlite using
[ImHex](https://github.com/WerWolv/ImHex) and the differences seem to
be:
1. SQLite uses varint according to [the
spec](https://www.sqlite.org/fileformat.html#record_format), whereas
limbo always encodes integers as i64
2. Limbo adds 4 bytes of zeros for overflow page pointer (even in cases
where the cell doesnt overflow)
3. Limbo adds a space after `CREATE TABLE name` before the `(` even when
user doesn't specify it?
I implemented the following:
- Fix 1: Varint serialization of i8, i16, i24, i32, i48 and i64
according to payload, instead of always using i64
- Fix 2: Removed the 4 bytes reserved for overflow page pointer in non-
overflow cases
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#550
small follow up to https://github.com/tursodatabase/limbo/pull/539
contains:
- Variable renaming and comments to `btreecursor.insert_into_cell()`
- New utility methods `pagecontent.header_size()`,
`pagecontent.cell_pointer_array_size()`,
`pagecontent.unallocated_region_start()` and
`pagecontent.unallocated_region_size()`
- Refactor of `btreecursor.compute_free_space()` (plus comments and
variable renaming)
- Rename `pagecontent.cell_get_raw_pointer_region()` to
`pagecontent.cell_pointer_array_offset_and_size()` and remove its usage
in `btreecursor.defragment_page()`
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#543
