We currently have two value types, `Value` and `OwnedValue`. The
original thinking was that `Value` is external type and `OwnedValue` is
internal type. However, this just results in unnecessary transformation
between the types as data crosses the Limbo library boundary.
Let's just follow SQLite here and consolidate on a single value type
(where `sqlite3_value` is just an alias for the internal `Mem` type).
The way this will eventually work is that we can have bunch of
pre-allocated `OwnedValue` objects in `ProgramState` and basically
return a reference to them all the way to the application itself, which
extracts the actual value.
```console
thread 'fuzz::tests::logical_expression_fuzz_run' panicked at tests\integration\fuzz\mod.rs:818:13:
assertion `left == right` failed: query: SELECT ( ( 3622873 || -8851250 ) * ( ( ( -124 ) + ( -5792536 ) ) ) ) = ( 179434259456392 < 65481085924370 ), limbo: [[Integer(1)]], sqlite: [[Integer(0)]]
left: [[Integer(1)]]
right: [[Integer(0)]]
```
This and a few other failing fuzzing tests were due to incorrectly
parsing numerics from strings. Some of our casting was done properly,
but it wasn't being applied to all cases where the behavior was needed.
It was also attempting to parse a string[0..N] N times until
`string[0..N].parse()` would no longer succeed. This searches for the
index of the first illegal character and parses the resulting slice
once.
Tests were added for some of the edgecases that were previously failing.
This PR also adds a macro in vdbe/insn.rs that allows for a bit of
cleanup and reduces some matching.
Closes#1053
This PR started out as one to improve the API of extensions but I ended
up building on top of this quite a bit and it just kept going. Sorry
this one is so large but there wasn't really a good stopping point, as
it kept leaving stuff in broken states.
**VCreate**: Support for `CREATE VIRTUAL TABLE t USING vtab_module`
**VUpdate**: Support for `INSERT` and `DELETE` methods on virtual
tables.
Sqlite uses `xUpdate` function with the `VUpdate` opcode to handle all
insert/update/delete functionality in virtual tables..
have to just document that:
```
if args[0] == NULL: INSERT args[1] the values in args[2..]
if args[1] == NULL: DELETE args[0]
if args[0] != NULL && len(args) > 2: Update values=args[2..] rowid=args[0]
```
I know I asked @jussisaurio on discord about this already, but it just
sucked so bad that I added some internal translation so we could expose
a [nice API](https://github.com/tursodatabase/limbo/pull/996/files#diff-
3e8f8a660b11786745b48b528222d11671e9f19fa00a032a4eefb5412e8200d1R54) and
handle the logic ourselves while keeping with sqlite's opcodes.
I'll change it back if I have to, I just thought it was genuinely awful
to have to rely on comments to explain all that to extension authors.
The included extension is not meant to be a legitimately useful one, it
is there for testing purposes. I did something similar in #960 using a
test extension, so I figure when they are both merged, I will go back
and combine them into one since you can do many kinds at once, and that
way it will reduce the amount of crates and therefore compile time.
1. Remaining opcodes.
2. `UPDATE` (when we support the syntax)
3. `xConnect` - expose API for a DB connection to a vtab so it can
perform arbitrary queries.
Closes#996
This PR fixes
[#1040](https://github.com/tursodatabase/limbo/issues/1040) and modifies
the `LIKE` function in the VDBE to work on expressions of all types like
SQLite.
Looking at how SQLite handles this, it gets the text value of the
expression regardless of its affinity. I used `exec_cast(exp, "TEXT")`
to achieve the same effect. Since most `LIKE` queries will probably be
done on `TEXT` expressions, I avoid casting the expression if it's
already `TEXT`. If either of the expressions was `NULL`, SQLite returns
nothing i.e. `NULL`. I also changed the unreachable arm message from
`Like on non-text registers` to `Like failed`.
The following queries produced the same results in Limbo:
```
SQLite version 3.46.1 2024-08-13 09:16:08 (UTF-16 console I/O)
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> CREATE TABLE tbl (n NULL, i INTEGER, r REAL, t TEXT, b BLOB);
sqlite> INSERT INTO tbl VALUES(NULL,1,2.0,'a',X'0500');
sqlite> SELECT * FROM tbl;
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE n LIKE NULL;
sqlite> SELECT * FROM tbl WHERE n LIKE 'NULL';
sqlite> SELECT * FROM tbl WHERE n LIKE 1;
sqlite> SELECT * FROM tbl WHERE n LIKE 2.0;
sqlite> SELECT * FROM tbl WHERE n LIKE x'0500';
sqlite>
sqlite> SELECT * FROM tbl WHERE i LIKE NULL;
sqlite> SELECT * FROM tbl WHERE i LIKE 1;
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE i LIKE '1';
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE i LIKE 2.0;
sqlite> SELECT * FROM tbl WHERE i LIKE 1.0;
sqlite> SELECT * FROM tbl WHERE i LIKE x'0500';
sqlite>
sqlite> SELECT * FROM tbl WHERE r LIKE NULL;
sqlite> SELECT * FROM tbl WHERE r LIKE 2;
sqlite> SELECT * FROM tbl WHERE r LIKE 2.0;
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE r LIKE '2.0';
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE r LIKE 'a';
sqlite> SELECT * FROM tbl WHERE r LIKE x'0500';
sqlite>
sqlite> SELECT * FROM tbl WHERE t LIKE NULL;
sqlite> SELECT * FROM tbl WHERE t LIKE 1;
sqlite> SELECT * FROM tbl WHERE t LIKE 2.0;
sqlite> SELECT * FROM tbl WHERE t LIKE 'a';
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE t LIKE x'0500';
sqlite>
sqlite> SELECT * FROM tbl WHERE b LIKE NULL;
sqlite> SELECT * FROM tbl WHERE b LIKE 1;
sqlite> SELECT * FROM tbl WHERE b LIKE 2.0;
sqlite> SELECT * FROM tbl WHERE b LIKE 'a';
sqlite> SELECT * FROM tbl WHERE b LIKE x'0500';
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE b LIKE 'x''0500''';
sqlite> SELECT * FROM tbl WHERE b LIKE '♣';
sqlite>
sqlite> SELECT * FROM tbl WHERE 1 LIKE 1;
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE 2.0 LIKE 2.0;
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE 2.0 LIKE '2.0';
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE '2.0' LIKE 2.0;
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE '123.45' LIKE 123.45;
|1|2.0|a|♣
sqlite> SELECT * FROM tbl WHERE NULL LIKE NULL;
sqlite> SELECT * FROM tbl WHERE x'0500' LIKE x'0500';
|1|2.0|a|♣
sqlite> SELECT typeof(n), typeof(i), typeof(r), typeof(t), typeof(b) FROM tbl;
null|integer|real|text|blob
```
Though, these queries are very basic, and more testing could be done.
Closes#1044
the command for drop table translation has been updated so that it more closely matches the semantics of SQLite's drop table command.
there are a few more things missing like ephemeral tables, destroy etc.
this is the initial commit is for the implementation of DROP TABLE. It adds support for the DROP TABLE instruction and adds a DropBTree instruction. It also implements the btree_drop method in btree.rs which makes use of free_page method which will be implemented via PR https://github.com/tursodatabase/limbo/pull/785
This is a first step to supporting [sqlite column
affinity](https://www.sqlite.org/datatype3.html) properly in limbo --
just adds an `affinity()` function to `Column` that can then be used
elsewhere
Closes#1030
Align `substr` implementation with SQLite spec
(https://www.sqlite.org/lang_corefunc.html#substr):
> The substr(X,Y,Z) function returns a substring of input string X that
begins with the Y-th character and which is Z characters long. If Z is
omitted then substr(X,Y) returns all characters through the end of the
string X beginning with the Y-th. The left-most character of X is number
1. If Y is negative then the first character of the substring is found
by counting from the right rather than the left. If Z is negative then
the abs(Z) characters preceding the Y-th character are returned. If X is
a string then characters indices refer to actual UTF-8 characters. If X
is a BLOB then the indices refer to bytes.
Reviewed-by: Jussi Saurio (@jussisaurio)
Closes#1013
According to SQLite documentation, the way to use these instructions
is to compare the seek key to the index key as you would with the
Compare opcode. The compare opcode states:
"Compare two vectors of registers in reg(P1)..reg(P1+P3-1)
(call this vector "A") and in reg(P2)..reg(P2+P3-1) ("B")."
In other words, we should compare the same number of columns from each,
not compare the entire keys.
This fixes a few Clickbench queries returning incorrect results, and
so closes#1009
---
Future work: support index seek keys that use multiple columns. Our
index seek is many times slower than SQLite because we're not utilizing
all the possible columns -- instead we just use the first index column
to seek.
This is an attempt to move towards #881. I am not sure this is the
direction you want to take. In any case, I thought I would take a crack
at converting `values` from `Record` to private and see how bad it would
be.
In the end, as you can see, it is not so bad. I think performance-wise
it shouldn't be a bad hit with Rust's zero-cost abstraction. Also,
during the process I noticed a couple improvements that could be made
here and there but I honestly wanted to start with something small
enough that wouldn't be too hard to review.
Anyway, let me know if this is really how you would like to proceed.
Closes#962
This PR aims to simplify text creation and to reduce allocation overhead
of creating a new OwnedValue::Text. Instead of creating Rc<String> every
time you need to create a text, you just pass the string slice and the
Rc<String> is created at the end. This change, at least on my machine,
has removed a lot of variability in the benchmarking performance, while
maintaining roughly the same performance.
Closes#961
Add fuzz test for string functions and fix 2 bugs in implementation of
`LTRIM/RTRIM/TRIM` and `QUOTE`:
- `QUOTE` needs to escape internal quote(`'`) symbols
- `LTRIM`/`RTRIM`/`TRIM` needs to check if they have additional argument
Closes#958
Fix codegen for `CAST` expression and also adjust implementation of
`CAST: Real -> Int` because `SQLite` use "closest integer between the
REAL value and zero" as a CAST result.
Closes#956
