### Feat:
- Adds support for descending indexes
### Testing:
- Augments existing compound key index seek fuzz test to test for
various combinations of ascending and descending indexed columns. To
illustrate, the test runs 10000 queries like this on 8 different tables
with all different asc/desc permutations of a three-column primary key
index:
```sql
query: SELECT * FROM t WHERE x = 2826 LIMIT 5
query: SELECT * FROM t WHERE x = 671 AND y >= 2447 ORDER BY x ASC, y DESC LIMIT 5
query: SELECT * FROM t WHERE x = 2412 AND y = 589 AND z >= 894 ORDER BY x DESC LIMIT 5
query: SELECT * FROM t WHERE x = 1217 AND y = 1437 AND z <= 265 ORDER BY x ASC, y ASC, z DESC LIMIT 5
query: SELECT * FROM t WHERE x < 138 ORDER BY x DESC LIMIT 5
query: SELECT * FROM t WHERE x = 1312 AND y = 2757 AND z > 39 ORDER BY x DESC, y ASC, z ASC LIMIT 5
query: SELECT * FROM t WHERE x = 1829 AND y >= 1629 ORDER BY x ASC, y ASC LIMIT 5
query: SELECT * FROM t WHERE x = 2047 ORDER BY x DESC LIMIT 5
query: SELECT * FROM t WHERE x = 893 AND y > 432 ORDER BY y DESC LIMIT 5
query: SELECT * FROM t WHERE x = 1865 AND y = 784 AND z <= 785 ORDER BY x DESC, y DESC, z DESC LIMIT 5
query: SELECT * FROM t WHERE x = 213 AND y = 1475 AND z <= 2870 ORDER BY x ASC, y ASC, z ASC LIMIT 5
query: SELECT * FROM t WHERE x >= 1780 ORDER BY x ASC LIMIT 5
query: SELECT * FROM t WHERE x = 1983 AND y = 602 AND z = 485 ORDER BY y ASC, z ASC LIMIT 5
query: SELECT * FROM t WHERE x = 2311 AND y >= 31 ORDER BY y DESC LIMIT 5
query: SELECT * FROM t WHERE x = 81 AND y >= 1037 ORDER BY x ASC, y DESC LIMIT 5
query: SELECT * FROM t WHERE x < 2698 ORDER BY x ASC LIMIT 5
query: SELECT * FROM t WHERE x = 1503 AND y = 554 AND z >= 185 ORDER BY x DESC, y DESC, z DESC LIMIT 5
query: SELECT * FROM t WHERE x = 619 AND y > 1414 ORDER BY x DESC, y ASC LIMIT 5
query: SELECT * FROM t WHERE x >= 865 ORDER BY x DESC LIMIT 5
query: SELECT * FROM t WHERE x = 1596 AND y = 622 AND z = 62 ORDER BY x DESC, z ASC LIMIT 5
query: SELECT * FROM t WHERE x = 1555 AND y = 1257 AND z < 1929 ORDER BY x ASC, y ASC, z ASC LIMIT 5
query: SELECT * FROM t WHERE x > 2598 LIMIT 5
query: SELECT * FROM t WHERE x = 302 AND y = 2476 AND z < 2302 ORDER BY z DESC LIMIT 5
query: SELECT * FROM t WHERE x = 2197 AND y = 2195 AND z > 2089 ORDER BY y ASC, z DESC LIMIT 5
query: SELECT * FROM t WHERE x = 1030 AND y = 1717 AND z < 987 LIMIT 5
query: SELECT * FROM t WHERE x = 2899 AND y >= 382 ORDER BY y DESC LIMIT 5
query: SELECT * FROM t WHERE x = 62 AND y = 2980 AND z < 1109 ORDER BY x DESC, y DESC, z DESC LIMIT 5
query: SELECT * FROM t WHERE x = 550 AND y > 221 ORDER BY y DESC LIMIT 5
query: SELECT * FROM t WHERE x = 376 AND y = 1874 AND z < 206 ORDER BY y DESC, z ASC LIMIT 5
query: SELECT * FROM t WHERE x = 859 AND y = 2157 ORDER BY x DESC LIMIT 5
query: SELECT * FROM t WHERE x = 2166 AND y = 2079 AND z < 301 ORDER BY x DESC, y ASC LIMIT 5
```
the queries are run against both sqlite and limbo.
Reviewed-by: Pere Diaz Bou (@pereman2)
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#1330
First attempt at closing #1212. Also with this PR, I added the option of
using `with` syntax for `TestLimboShell`. It automatically closes the
shell on error, and facilitates error handling overall. If this is
merged, I can update the other python tests to use `with` as well.
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#1230
Some more helper utilities extracted out from #1351 to make the surface
area of that PR smaller
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#1353
Apply affinities to a range of P2 registers starting with P1.
P4 is a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th memory cell in the range.
If P4==0 then register P3 holds a blob constructed by MakeRecord. If P4>0 then register P3 is the first of P4 registers that form an unpacked record.
Cursor P1 is on an index btree. If the record identified by P3 and P4 is not the prefix of any entry in P1 then a jump is made to P2. If P1 does contain an entry whose prefix matches the P3/P4 record then control falls through to the next instruction and P1 is left pointing at the matching entry.
This operation leaves the cursor in a state where it cannot be advanced in either direction. In other words, the Next and Prev opcodes do not work after this operation.
When the python tests fail, they will sometimes truncate the output if
it is smaller than the `PIPE_BUFF` size. With this fix we can now
properly print the backtrace, when the program panics.
# Before
This is the problematic CI output from #1331 that led me to fix this.
In this case, it was already truncating the output of the `assert`
prints.
```
./testing/cli_tests/extensions.py
Extension ./target/debug/liblimbo_regexp loaded successfully.
Testing: uuid functions are registered properly with ext loaded
Testing: scalar alias's are registered properly
Testing: median agg function returns null when ext not loaded
Testing: median agg function works
Testing: median agg function works with odd number of elements
Testing: test aggregate percentile function with 2 arguments works
Testing: test aggregate percentile function with 1 argument works
Testing: crypto_blake3 returns null when ext not loaded
Testing: blake3 should encrypt correctly
Testing: md5 should encrypt correctly
Testing: sha1 should encrypt correctly
Testing: sha256 should encrypt correctly
Testing: sha384 should encrypt correctly
Testing: sha512 should encrypt correctly
Testing: base32 should encode correctly
Testing: base32 should decode correctly
Testing: base64 should encode correctly
Testing: base64 should decode correctly
Testing: base85 should encode correctly
Testing: base85 should decode correctly
Testing: hex should encode correctly
Testing: hex should decode correctly
Testing: url should encode correctly
Testing: url should decode correctly
Testing: ipfamily function returns null when ext not loaded
Testing: ipfamily function returns 4 for IPv4
Testing: ipfamily function returns 6 for IPv6
Testing: ipcontains function returns 1 for IPv4
Testing: ipcontains function returns 0 for IPv4
Testing: iphost function returns the host for IPv4
Testing: iphost function returns the host for IPv6
Testing: ipmasklen function returns the mask length for IPv4
Testing: ipmasklen function returns the mask length for IPv6
Testing: ipnetwork function returns the flattened CIDR for IPv4
Testing: ipnetwork function returns the network for IPv6
Testing: testvfs not loaded
Testing: testvfs extension loaded
thread 'main' panicked at core/storage/pager.rs:61:38:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Test FAILED: Error encountered in Limbo shell.
make: *** [Makefile:70: test-extensions] Error 1
```
# After
```
with-env {RUST_BACKTRACE:1} {make test-extensions}
cargo build
Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.51s
cargo build --package limbo_regexp
Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.19s
./testing/cli_tests/extensions.py
Extension ./target/debug/liblimbo_regexp loaded successfully.
Testing: uuid functions are registered properly with ext loaded
Testing: scalar alias's are registered properly
Testing: median agg function returns null when ext not loaded
Testing: median agg function works
Testing: median agg function works with odd number of elements
Testing: test aggregate percentile function with 2 arguments works
Testing: test aggregate percentile function with 1 argument works
Testing: crypto_blake3 returns null when ext not loaded
Testing: blake3 should encrypt correctly
Testing: md5 should encrypt correctly
Testing: sha1 should encrypt correctly
Testing: sha256 should encrypt correctly
Testing: sha384 should encrypt correctly
Testing: sha512 should encrypt correctly
Testing: base32 should encode correctly
Testing: base32 should decode correctly
Testing: base64 should encode correctly
Testing: base64 should decode correctly
Testing: base85 should encode correctly
Testing: base85 should decode correctly
Testing: hex should encode correctly
Testing: hex should decode correctly
Testing: url should encode correctly
Testing: url should decode correctly
Testing: ipfamily function returns null when ext not loaded
Testing: ipfamily function returns 4 for IPv4
Testing: ipfamily function returns 6 for IPv6
Testing: ipcontains function returns 1 for IPv4
Testing: ipcontains function returns 0 for IPv4
Testing: iphost function returns the host for IPv4
Testing: iphost function returns the host for IPv6
Testing: ipmasklen function returns the mask length for IPv4
Testing: ipmasklen function returns the mask length for IPv6
Testing: ipnetwork function returns the flattened CIDR for IPv4
Testing: ipnetwork function returns the network for IPv6
Testing: testvfs not loaded
Testing: testvfs extension loaded
thread 'main' panicked at core/storage/pager.rs:61:38:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: rust_begin_unwind
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
2: core::panicking::panic
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:148:5
3: core::option::unwrap_failed
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/option.rs:2012:5
4: core::option::Option<T>::unwrap
at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:972:21
5: limbo_core::storage:📟:Page::get_contents
at ./core/storage/pager.rs:61:9
6: limbo_core::storage::btree::BTreeCursor::balance_non_root
at ./core/storage/btree.rs:1723:40
7: limbo_core::storage::btree::BTreeCursor::balance
at ./core/storage/btree.rs:1570:35
8: limbo_core::storage::btree::BTreeCursor::insert_into_page
at ./core/storage/btree.rs:1512:35
9: limbo_core::storage::btree::BTreeCursor::insert
at ./core/storage/btree.rs:3024:31
10: limbo_core::vdbe::execute::op_insert
at ./core/vdbe/execute.rs:3654:23
11: limbo_core::vdbe::Program::step
at ./core/vdbe/mod.rs:379:23
12: limbo_core::Statement::step
at ./core/lib.rs:582:9
13: limbo::app::Limbo::print_query_result
at ./cli/app.rs:657:27
14: limbo::app::Limbo::run_query
at ./cli/app.rs:420:20
15: limbo::app::Limbo::handle_input_line
at ./cli/app.rs:527:13
16: limbo::main
at ./cli/main.rs:29:31
17: core::ops::function::FnOnce::call_once
at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Testing: Tested large write to testfs
Test FAILED: Test failed
SQL: SELECT count(*) FROM test;
Actual:
None
make: *** [test-extensions] Error 1
```
Closes#1346
Sqlite reference implementation: https://github.com/sqlite/sqlite/blob/8
37dc09bce7de8971c7488b70cf5da93c60fbed0/src/vdbe.c#L2558
We did not support blobs before and our ordering of the match statements
were incorrect when one of the arguments was NULL
Closes#1337
Currently, to run the fuzz test you need to remove an assert statement.
EDIT: @diegoreis42 pointed it out to me that the method had just changed
actually. Made the change to use `row.len()` instead
# Problem
```
cargo +nightly fuzz run --target=aarch64-apple-darwin expression
Compiling limbo_core v0.0.19-pre.4 (/Users/pedro/Projects/limbo/core)
Compiling limbo-fuzz v0.0.0 (/Users/pedro/Projects/limbo/fuzz)
error[E0599]: `&limbo_core::vdbe::Row` is not an iterator
--> fuzz_targets/expression.rs:198:36
|
198 | assert_eq!(row.count(), 1, "expr: {:?}", expr);
| ^^^^^
| |
| `&limbo_core::vdbe::Row` is not an iterator
| private field, not a method
|
::: /Users/pedro/Projects/limbo/core/vdbe/mod.rs:221:1
|
221 | pub struct Row {
| -------------- doesn't satisfy `limbo_core::vdbe::Row: std::iter::Iterator`
|
= note: the following trait bounds were not satisfied:
`&limbo_core::vdbe::Row: std::iter::Iterator`
which is required by `&mut &limbo_core::vdbe::Row: std::iter::Iterator`
`limbo_core::vdbe::Row: std::iter::Iterator`
which is required by `&mut limbo_core::vdbe::Row: std::iter::Iterator`
For more information about this error, try `rustc --explain E0599`.
error: could not compile `limbo-fuzz` (bin "expression") due to 1 previous error
Error: failed to build fuzz script: ASAN_OPTIONS="detect_odr_violation=0" RUSTFLAGS="-Cpasses=sancov-module -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-inline-8bit-counters -Cllvm-args=-sanitizer-coverage-pc-table -Cllvm-args=-sanitizer-coverage-trace-compares --cfg fuzzing -Clink-dead-code -Zsanitizer=address -Cdebug-assertions -C codegen-units=1" "cargo" "build" "--manifest-path" "/Users/pedro/Projects/limbo/fuzz/Cargo.toml" "--target" "aarch64-apple-darwin" "--release" "--config" "profile.release.debug=true" "--bin" "expression"
```
Closes#1334
Fixes#1298
- Fixes Limbo trying to use an index using a WHERE clause constraint
that refers to the same table on both sides, e.g. `WHERE t.x = t.x`
- Fixes not using indexes when the relevant expression is paren wrapped,
e.g.
- `SELECT * FROM t WHERE (indexcol) > 5`
- `SELECT * FROM t WHERE (indexcol > 5)`
- Changes existing table logical expr fuzz test to have primary keys
(which helped me find both issues above)
Closes#1300
Fixed is_empty assertion in #1203 , but simulator still has error.
```sh
[ERROR] error Internal error: select '(engrossing_berger < X'6566651A3C70278D4E200657551D8071A1' AND competitive_petit > 1236742147.9451914)' should return no values for table 'super_becky'
```
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#1308
