This PR implements more sophisticated algorithm in the toy vector sparse
index: now we enumerate components based on the frequency (in order to
check unpopular "features" first) and also estimate length threshold
which can give us better results compared with current top-k set.
Also, this PR adds optional `delta` parameter which can enable
approximate search which will return results with score not more than
`delta` away from the optimal.
In order to implement this index method - index code were slightly
adjusted in order to allow to store some non-key payload in the index
rows. So, now index can hold N columns where first K <= N columns will
be used as identity (before that K always was equal to N).
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#3862
Partial sync for sync engine will need to implement its own version of
`DatabaseStorage` which willl load database pages on demand
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3922
## Related issue
- closes#3885
## Description
Add a check to reject dropping a table when PRAGMA foreign_keys=ON and
the table is referenced by foreign keys
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3913
Closes#3892Closes#3888
Stuff like:
```sql
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select case (select y, z from t2) when 1 then 'one' else 'other' end from t1;
× Parse error: base expression in CASE must return 1 value
turso> create table t(x, y);
insert into t values (1, 2);
select (select x, y from t) as result;
× Parse error: result column must return 1 value, got 2
turso> create table t1(x,y);
create table t2(y);
insert into t1 values (1,1);
insert into t2 values (1);
select * from t2 where y = (select x,y from t1);
× Parse error: all arguments to binary operator = must return the same number of
│ values. Got: (1) = (2)
turso> create table orders(customer_id, amount);
create table thresholds(min_amount, max_amount);
insert into orders values (100, 50), (100, 150);
insert into thresholds values (100, 200);
select customer_id, sum(amount) as total
from orders
group by customer_id
having total > (select min_amount, max_amount from thresholds);
× Parse error: all arguments to binary operator > must return the same number of
│ values. Got: (1) > (2)
turso> create table items(id);
create table config(max_results, other_col);
insert into items values (1), (2), (3);
insert into config values (2, 3);
select * from items limit (select max_results, other_col from config);
× Parse error: limit expression must return 1 value, got 2
turso> create table items(id);
create table config(skip_count, other_col);
insert into items values (1), (2), (3);
insert into config values (1, 2);
select * from items limit 1 offset (select skip_count, other_col from config);
× Parse error: offset expression must return 1 value, got 2
turso> create table items(id, name);
create table sort_order(priority, other_col);
insert into items values (1, 'a'), (2, 'b');
insert into sort_order values (1, 2);
select * from items order by (select priority, other_col from sort_order);
× Parse error: order by expression must return 1 value, got 2
turso> create table sales(product_id, amount);
create table grouping(category, other_col);
insert into sales values (1, 100), (2, 200);
insert into grouping values (1, 2);
select sum(amount) from sales group by (select category, other_col from grouping);
× Parse error: group by expression must return 1 value, got 2
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select case when (select y, z from t2) then 'yes' else 'no' end from t1;
× Parse error: when expression in CASE must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select case when x = 1 then (select y, z from t2) else 0 end from t1;
× Parse error: then expression in CASE must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select case when x = 2 then 0 else (select y, z from t2) end from t1;
× Parse error: else expression in CASE must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select max((select y, z from t2)) from t1;
× Parse error: argument 0 to function call max must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select x + (select y, z from t2) from t1;
× Parse error: all arguments to binary operator + must return the same number of
│ values. Got: (1) + (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (5);
insert into t2 values (1, 2);
select * from t1 where x between (select y, z from t2) and 10;
× Parse error: all arguments to binary operator <= must return the same number of
│ values. Got: (2) <= (1)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select cast((select y, z from t2) as integer) from t1;
× Parse error: argument to CAST must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values ('a', 'b');
select (select y, z from t2) collate nocase from t1;
× Parse error: argument to COLLATE must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select * from t1 where (select y, z from t2) is null;
× Parse error: all arguments to binary operator IS must return the same number of
│ values. Got: (2) IS (1)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select * from t1 where (select y, z from t2) not null;
× Parse error: argument to NOT NULL must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values ('a', 'b');
select * from t1 where (select y, z from t2) like 'a%';
× Parse error: left operand of LIKE must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select -(select y, z from t2) from t1;
× Parse error: argument to unary operator - must return 1 value. Got: (2)
turso> create table t1(x);
create table t2(y, z);
insert into t1 values (1);
insert into t2 values (1, 2);
select abs((select y, z from t2)) from t1;
× Parse error: argument 0 to function call abs must return 1 value. Got: (2)
```
Closes#3906
closes https://github.com/tursodatabase/turso/issues/3773
## Before
```rust
#[derive(Debug, Clone)]
pub struct Column {
pub name: Option<String>,
pub ty: Type,
// many sqlite operations like table_info retain the original string
pub ty_str: String,
pub primary_key: bool,
pub is_rowid_alias: bool,
pub notnull: bool,
pub default: Option<Box<Expr>>,
pub unique: bool,
pub collation: Option<CollationSeq>,
pub hidden: bool,
}
```
obviously not ideal. so lets pack `type`, `hidden`, `primary_key`,
`is_rowid_alias`, `notnull` and `collation` into a u16.
## After:
```rust
#[derive(Debug, Clone)]
pub struct Column {
pub name: Option<String>,
pub ty_str: String,
pub default: Option<Box<Expr>>,
raw: u16,
}
```
Also saw a place to replace a `Mutex<Enum>` with `AtomicEnum`, so I
snuck that in here too
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#3905
This PR fixes sync engine bug which leads to the state of db going back
in time.
The mistake was made in the pull operation which before fetched
information about last commited changes to the remote separately. This
crates a problem since pull already works with fixed WAL updates
received earlier from remote - and this WAL update can be inconsistent
with more fresh value of last_change_id fetched from remote.
The fix is to use only WAL update and "extract" necessary information
from it. In order to do that sync now read meta sync table while pull
operation is in progress (at the moment when local changes are rolled
back and remote changes already applied) and do not use any external
source to consume that information.
Also, this PR fixes bug in the JS tursodatabase client and reset
statement in the finally block opposed to the previous approach to reset
statement at the beginning. The problem with previous approach were in
cases when client do not fully consumed the statement (e.g. abort
iteration and take only one row) in which case the statement will be
kept active and can prevent another write transaction from starting or
just occupy place as a read transaction.
Closes#3860
This PR reset cursor state in the `stmt.reset()` method because under
the hood statement caches some BTree state which can be no longer valid
at the moment of next statement run.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3859
