Commit Graph

924 Commits

Author SHA1 Message Date
Pekka Enberg
d4cc5151fb Merge 'github: Fix stale workflow' from Pekka Enberg
Bump operations per run but more importantly turn "ascending" option so
that we process pull requests with smaller IDs first that are more
likely to be stale.

Closes #337
2024-09-21 08:54:07 +03:00
Pekka Enberg
8380c0740b github: Fix stale workflow
Bump operations per run but more importantly turn "ascending" option so
that we process pull requests with smaller IDs first that are more
likely to be stale.
2024-09-21 08:20:42 +03:00
Pekka Enberg
b67640c437 Merge 'core: constraint check uniqueness' from Pere Diaz Bou
```bash
limbo> insert into products values (1, 'asdf', 432);
Runtime error: UNIQUE constraint failed: products.id (19)
```

Closes #336
2024-09-20 13:27:10 +03:00
Pere Diaz Bou
c8e5673f7a core: constraint check uniqueness 2024-09-20 11:45:55 +02:00
Pekka Enberg
c57e971eb3 Merge 'Add support for sqlite_version() scalar function ' from Kim Seon Woo
### sqlite
<img width="792" alt="image" src="https://github.com/user-
attachments/assets/1a9238db-d948-4583-a808-f9adfec7c534">
### limbo
<img width="809" alt="image" src="https://github.com/user-
attachments/assets/ea3e5f7e-bb3e-450d-be34-59ca00128beb">
### Changes
- Add support for `sqlite_version()` function
- Update function's explain message depending on the number of arguments

Closes #335
2024-09-18 13:50:51 +03:00
김선우
f7304a15a6 Fix clippy 2024-09-18 17:59:11 +09:00
김선우
e9ba458514 Add support for sqlite_version 2024-09-18 17:57:51 +09:00
Pekka Enberg
ac6dd71de6 Update CHANGELOG.md 2024-09-18 09:47:05 +03:00
Pekka Enberg
a8b8c1d2c9 Merge 'Fix clippy ' from Kim Seon Woo
### Changes
- Remove clippy warning messages
- Add `#[allow(clippy...)]` in places where it might be better not to
fix
### TODO
recommended changes by `cargo fmt` on my local and github differs on
`sqlite3/src/lib.rs`. Should check for the reason
=> just upgrade the rust toolchain

Closes #329
2024-09-18 09:46:07 +03:00
Pekka Enberg
e364d10d9a Merge 'Implementation of scalar function glob() and GLOB operator' from RJ Barman
Related to #144
- Separates `glob` and `like` regex caches, since the same pattern would
result in a different regex depending on which rules you apply
- Also fixes the value of `constant_mask` when translating LIKE
expressions now that non-constant pattern values are supported.
Note that LIKE and GLOB are almost entirely the same, the only
difference being the matching rules (so in our case, the regex
construction).

Closes #334
2024-09-18 09:44:51 +03:00
Pekka Enberg
8bc467bca8 Update CHANGELOG.md 2024-09-18 09:44:25 +03:00
Pekka Enberg
35032395e7 Merge 'Make LIKE expressions case-insensitive' from RJ Barman
Per the sqlite docs, LIKE is case-insensitive.
> The LIKE operator does a pattern matching comparison. The operand to
the right of the LIKE operator contains the pattern and the left hand
operand contains the string to match against the pattern. A percent
symbol ("%") in the LIKE pattern matches any sequence of zero or more
characters in the string. An underscore ("_") in the LIKE pattern
matches any single character in the string. **Any other character
matches itself or its lower/upper case equivalent (i.e. case-insensitive
matching)**. Important Note: SQLite only understands upper/lower case
for ASCII characters by default. The LIKE operator is case sensitive by
default for unicode characters that are beyond the ASCII range. For
example, the expression 'a' LIKE 'A' is TRUE but 'æ' LIKE 'Æ' is FALSE.
The ICU extension to SQLite includes an enhanced version of the LIKE
operator that does case folding across all unicode characters.
Note that sqlite does not support case-insensitive comparisons of
unicode characters by default. This PR _does not_ match that behavior
currently. I can change things to not support unicode-case-
insensitivity, but as I understand it, doing so may require disable the
unicode-case feature on the regex crate, and potentially using a
`regex::bytes::Regex`, instead of a `regex::Regex`. (Basically, doing
some stuff that would not match anyone's initial assumptions about how
this would work).

Closes #333
2024-09-18 09:44:00 +03:00
rjhallsted
f6b7a3fba4 Remove duplication of glob and like translation 2024-09-16 16:31:37 -07:00
rjhallsted
338effa0ca Separate glob and like regex caches 2024-09-16 16:22:31 -07:00
rjhallsted
af09f0b7f4 Add testing 2024-09-16 16:18:26 -07:00
rjhallsted
f39dc534a6 add glob execution 2024-09-16 16:12:53 -07:00
rjhallsted
b87b874ed0 WIP commit on glob support 2024-09-16 15:51:32 -07:00
rjhallsted
4aaa0b0989 Make LIKE expressions case-insensitive 2024-09-16 14:43:39 -07:00
김선우
9e2756fabc Nit 2024-09-16 23:07:50 +09:00
김선우
905b34c9a8 Nit 2024-09-16 22:03:53 +09:00
김선우
7f5d9dbb90 Apply fmt 2024-09-16 22:00:36 +09:00
김선우
b68efa32d9 Merge branch 'main' into feature/fix-clippy 2024-09-16 21:58:07 +09:00
Pekka Enberg
19d2d92115 Merge 'test: Switch to bundled SQLite for rusqlite' from Pekka Enberg
We're seeing build errors on Windows so let's see if this fixes it.

Closes #332
2024-09-16 15:03:39 +03:00
Pekka Enberg
2760049a89 test: Ignore failing tests... 2024-09-16 14:43:43 +03:00
Pekka Enberg
f3ce6a91ba test: Switch to bundled SQLite for rusqlite
We're seeing build errors on Windows so let's see if this fixes it.
2024-09-16 14:29:13 +03:00
Pekka Enberg
9bbfdab5fa Revert "Merge 'Add support for sqlite_version() scalar function' from Kim Seon Woo"
This reverts commit e365c12ce0, reversing
changes made to 21bd1a961e. The pull request broke some tests:

```
thread 'main' panicked at core/vdbe/mod.rs:1713:72:
index out of bounds: the len is 3 but the index is 3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    while executing
"exec {*}$command"
    (procedure "evaluate_sql" line 3)
    invoked from within
"evaluate_sql $sqlite_exec $sql"
    (procedure "run_test" line 2)
    invoked from within
"run_test $::sqlite_exec $combined_sql $combined_expected_output"
    (procedure "do_execsql_test" line 5)
    invoked from within
"do_execsql_test sqlite_version {
  SELECT sqlite_version();
} {3.46.1}"
    (file "./testing/scalar-functions.test" line 434)
    invoked from within
"source $testdir/scalar-functions.test"
    (file "./testing/all.test" line 16)
make: *** [Makefile:59: test-compat] Error 1
```
2024-09-16 14:28:18 +03:00
Pekka Enberg
e365c12ce0 Merge 'Add support for sqlite_version() scalar function' from Kim Seon Woo
### sqlite
<img width="792" alt="image" src="https://github.com/user-
attachments/assets/1a9238db-d948-4583-a808-f9adfec7c534">
### limbo
<img width="809" alt="image" src="https://github.com/user-
attachments/assets/ea3e5f7e-bb3e-450d-be34-59ca00128beb">
### Changes
- Add support for `sqlite_version()` function
- Update function's explain message depending on the number of arguments

Closes #331
2024-09-16 14:18:19 +03:00
Pekka Enberg
21bd1a961e Merge 'Fix three issues with LIKE operator (#319)' from Jussi Saurio
Closes #319
1. Allow using a column as the pattern
2. Construct LIKE regexes with `^` and `$` so that eg the string
`'foobar'` does not match the pattern `'fooba'` unless the pattern
specifically has a wildcard
3. Support function expressions as the LIKE pattern

Closes #327
2024-09-16 14:17:48 +03:00
Pekka Enberg
95987f9824 Merge 'Support grouping by a function expression' from Jussi Saurio
Closes #326
2024-09-16 14:17:03 +03:00
김선우
469f345c16 Fix explain message for function depending on the number of args 2024-09-16 18:47:53 +09:00
김선우
6b40acabbc Add support for sqlite_version() scalar function 2024-09-16 18:38:42 +09:00
김선우
f592ed15c1 Add test to verify root and leaf splitting 2024-09-16 16:50:30 +09:00
김선우
2086d0e264 Revert max_iterations 2024-09-16 03:25:49 +09:00
jussisaurio
9f1ce53d18 Merge 'implementation of scalar function sign(X)' from Jean Arhancet
This is related to the issue
https://github.com/penberg/limbo/issues/144. Add the scalar function
`sign(X)`

Closes #328
2024-09-15 16:03:09 +03:00
JeanArhancet
d03a734f21 feat: add sign function 2024-09-15 14:57:01 +02:00
김선우
ff1c38e0bb Rewrite test to use temp dir 2024-09-15 17:31:55 +09:00
김선우
07c4ca9edd Nit 2024-09-15 17:16:33 +09:00
김선우
f08dc9e2c0 Revert sqlite3/src/lib 2024-09-15 16:32:12 +09:00
김선우
28884181be Fix clippy 2024-09-15 16:23:27 +09:00
jussisaurio
3f0162c76c Support grouping by a function expression 2024-09-15 09:58:46 +03:00
jussisaurio
6b8cd02f71 Fix function expressions in like pattern not working 2024-09-15 09:57:54 +03:00
jussisaurio
234c56ca81 Fix two issues with LIKE operator (#319) 2024-09-15 09:57:54 +03:00
jussisaurio
d4971a4706 Merge 'cargo clippy --fix --allow-dirty && cargo fmt' from Jussi Saurio
Closes #325
2024-09-15 09:56:02 +03:00
jussisaurio
b6e88ca883 cargo clippy --fix --allow-dirty && cargo fmt 2024-09-15 09:35:39 +03:00
jussisaurio
26d88a788b Merge 'Add support for concat_ws scalar function' from Kim Seon Woo
Adding support for `concat_ws`
<img width="1604" alt="image" src="https://github.com/user-
attachments/assets/8281d0c1-c338-4695-aa17-3b86f16625d3">
## Related issue
https://github.com/penberg/limbo/issues/144

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #324
2024-09-15 09:09:40 +03:00
김선우
8b3e4da8da Apply comments
- Specify the number of arguments using arg_count
- Add test for concat_ws
2024-09-15 10:04:30 +09:00
김선우
fc07ca9d73 Add support for concat_ws 2024-09-14 23:47:45 +09:00
Pekka Enberg
51e3fe487a Update CHANGELOG.md 2024-09-14 16:36:40 +03:00
Pekka Enberg
2395ed2b71 Merge 'core,btree: support overflow pages' from Pere Diaz Bou
This PR adds support for traversing overflow payloads and inserting them
too. The test added is a simple one and must be tested with higher
degree of stress in the future.

Closes #322
2024-09-14 16:35:02 +03:00
Pekka Enberg
f39120dbb1 Merge 'GROUP BY support' from Jussi Saurio
This PR implements GROUP BY (and also ordering by the groups or the
aggregate expressions). See `groupby.test` for the kinds of things that
are now supported.
This PR is a rabbit hole and insanely big, sorry.
---
I thought about how to explain how GROUP BY works in SQLite bytecode,
and opted to go for just adding a bunch of Insn comments, so here's an
example that uses both GROUP BY and ORDER BY:
**LIMBO**
```
limbo> explain select u.first_name, sum(u.age) from users u group by u.first_name order by sum(u.age);
addr  opcode             p1    p2    p3    p4             p5  comment
----  -----------------  ----  ----  ----  -------------  --  -------
0     Init               0     56    0                    0   Start at 56
1     SorterOpen         0     1     0     k(1,B)         0   cursor=0
2     SorterOpen         1     2     0     k(1,B)         0   cursor=1
3     Integer            0     2     0                    0   r[2]=0; clear group by abort flag
4     Null               0     4     0                    0   r[4]=NULL; initialize group by comparison registers to NULL
5     Gosub              8     45    0                    0   ; go to clear accumulator subroutine
6     OpenReadAsync      2     2     0                    0   table=u, root=2
7     OpenReadAwait      0     0     0                    0
8     RewindAsync        2     0     0                    0
9     RewindAwait        2     38    0                    0   Rewind table u
10      Column           2     1     10                   0   r[10]=u.first_name
11      Column           2     9     11                   0   r[11]=u.age
12      MakeRecord       10    2     7                    0   r[7]=mkrec(r[10..11])
13      SorterInsert     1     7     0     0              0   key=r[7]
14    NextAsync          2     0     0                    0
15    NextAwait          2     9     0                    0
16    OpenPseudo         3     7     2                    0   2 columns in r[7]
17    SorterSort         1     32    0                    0
18      SorterData       1     7     3                    0   r[7]=data
19      Column           3     0     12                   0   r[12]=cursor 3.u.first_name
20      Compare          4     12    1                    0   r[4..4]==r[12..12]
21      Jump             22    26    22                   0   ; start new group if comparison is not equal
22      Move             12    4     1                    0   r[4..4]=r[12..12]
23      Gosub            9     36    0                    0   ; check if ended group had data, and output if so
24      IfPos            2     55    0                    0   r[2]>0 -> r[2]-=0, goto 55; check abort flag
25      Gosub            8     45    0                    0   ; goto clear accumulator subroutine
26      Column           3     1     13                   0   r[13]=cursor 3.u.age
27      AggStep          0     13    6     sum            0   accum=r[6] step(r[13])
28      If               3     30    0                    0   if r[3] goto 30; don't emit group columns if continuing existing group
29      Column           3     0     5                    0   r[5]=cursor 3.u.first_name
30      Integer          1     3     0                    0   r[3]=1; indicate data in accumulator
31    SorterNext         1     18    0                    0
32    Gosub              9     36    0                    0   ; emit row for final group
33    Goto               0     48    0                    0   ; group by finished
34    Integer            1     2     0                    0   r[2]=1
35    Return             9     0     0                    0
36    IfPos              3     38    0                    0   r[3]>0 -> r[3]-=0, goto 38; output group by row subroutine start
37    Return             9     0     0                    0
38    AggFinal           0     6     0     sum            0   accum=r[6]
39    Copy               5     15    0                    0   r[15]=r[5]
40    Copy               6     16    0                    0   r[16]=r[6]
41    Copy               6     14    0                    0   r[14]=r[6]
42    MakeRecord         14    3     1                    0   r[1]=mkrec(r[14..16])
43    SorterInsert       0     1     0     0              0   key=r[1]
44    Return             9     0     0                    0
45    Null               0     5     6                    0   r[5..6]=NULL; clear accumulator subroutine start
46    Integer            0     3     0                    0   r[3]=0
47    Return             8     0     0                    0
48    OpenPseudo         4     1     3                    0   3 columns in r[1]
49    SorterSort         0     55    0                    0
50      SorterData       0     1     4                    0   r[1]=data
51      Column           4     1     17                   0   r[17]=cursor 4.sum
52      Column           4     2     18                   0   r[18]=cursor 4.u.first_name
53      ResultRow        17    2     0                    0   output=r[17..18]
54    SorterNext         0     50    0                    0
55    Halt               0     0     0                    0
56    Transaction        0     0     0                    0
57    Goto               0     1     0                    0
```
**SQLITE3**:
```
sqlite> explain select u.first_name, sum(u.age) from users u group by u.first_name order by sum(u.age);
addr  opcode         p1    p2    p3    p4             p5  comment
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     52    0                    0   Start at 52
1     SorterOpen     1     4     0     k(1,B)         0
2     SorterOpen     2     2     0     k(1,B)         0
3     Integer        0     2     0                    0   r[2]=0; clear abort flag
4     Null           0     5     5                    0   r[5..5]=NULL
5     Gosub          4     41    0                    0
6     OpenRead       0     2     0     10             0   root=2 iDb=0; users
7     Rewind         0     13    0                    0
8       Column         0     1     10                   0   r[10]= cursor 0 column 1
9       Column         0     9     11                   0   r[11]= cursor 0 column 9
10      MakeRecord     10    2     12                   0   r[12]=mkrec(r[10..11])
11      SorterInsert   2     12    0                    0   key=r[12]
12    Next           0     8     0                    1
13    OpenPseudo     3     12    2                    0   2 columns in r[12]
14    SorterSort     2     44    0                    0   GROUP BY sort
15      SorterData     2     12    3                    0   r[12]=data
16      Column         3     0     6                    0   r[6]= cursor 3 column 0
17      Compare        5     6     1     k(1,B)         0   r[5] <-> r[6]
18      Jump           19    23    19                   0
19      Move           6     5     1                    0   r[5]=r[6]
20      Gosub          3     33    0                    0   output one row
21      IfPos          2     44    0                    0   if r[2]>0 then r[2]-=0, goto 44; check abort flag
22      Gosub          4     41    0                    0   reset accumulator
23      Column         3     1     13                   0   r[13]=users.age
24      AggStep        0     13    9     sum(1)         1   accum=r[9] step(r[13])
25      If             1     27    0                    0
26      Column         3     0     7                    0   r[7]=users.first_name
27      Integer        1     1     0                    0   r[1]=1; indicate data in accumulator
28    SorterNext     2     15    0                    0
29    Gosub          3     33    0                    0   output final row
30    Goto           0     44    0                    0
31    Integer        1     2     0                    0   r[2]=1; set abort flag
32    Return         3     0     0                    0
33    IfPos          1     35    0                    0   if r[1]>0 then r[1]-=0, goto 35; Groupby result generator entry point
34    Return         3     0     0                    0
35    AggFinal       9     1     0     sum(1)         0   accum=r[9] N=1
36    Copy           7     15    0                    0   r[15]=r[7]
37    Copy           9     14    0                    0   r[14]=r[9]
38    MakeRecord     14    2     17                   0   r[17]=mkrec(r[14..15])
39    SorterInsert   1     17    14    2              0   key=r[17]
40    Return         3     0     0                    0   end groupby result generator
41    Null           0     7     9                    0   r[7..9]=NULL
42    Integer        0     1     0                    0   r[1]=0; indicate accumulator empty
43    Return         4     0     0                    0
44    OpenPseudo     4     18    4                    0   4 columns in r[18]
45    SorterSort     1     51    0                    0
46      SorterData     1     18    4                    0   r[18]=data
47      Column         4     0     16                   0   r[16]=sum(u.age)
48      Column         4     1     15                   0   r[15]=u.first_name
49      ResultRow      15    2     0                    0   output=r[15..16]
50    SorterNext     1     46    0                    0
51    Halt           0     0     0                    0
52    Transaction    0     0     2     0              1   usesStmtJournal=0
53    Goto           0     1     0                    0
```
As you can see the bytecodes are fairly close in this scenario.
SQLite opts to use an ephemeral index in certain cases (e.g. when you
use `LIMIT` or have multiple grouping columns). Will not implement those
branching strategies as part of this PR
---
Example operator tree:
```
limbo> explain query plan select u.first_name, p.name, sum(u.age) from users u join products p on u.id = p.id group by u.first_name, p.name order by p.name limit 10;

QUERY PLAN
`--TAKE 10
   `--SORT p.name ASC
   |  `--PROJECT u.first_name, p.name, sum (u.age)
   |  |  `--AGGREGATE Sum(u.age)
   |  |  |  `--JOIN
   |  |  |  |  |--SCAN users AS u
   |  |  |  |  `--SEEK products.rowid ON rowid=u.id
```

Reviewed-by: Pere Diaz Bou <pere-altea@hotmail.com>

Closes #309
2024-09-14 16:33:09 +03:00