SQLite does not allow us to modify system tables, but we do. Let's fix
it.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#2855
- otherwise, in multi-threading environment, other thread can think that
completion is finished and start execution
- this can lead to violated assertions (for example, page must be
loaded, but as callback is not executed yet assert will be fired)
Failing scenario:
1. main thread wants to execute pread - so it schedule IO and return
control to the caller
2. IO thread read data from the disk
3. IO thread executes complete(result)
4. complete func set result of the completion to Ok
5. main thread enter into the step loop again and check completion
status
6. completion marked as finished/is_completed - so main thread continue
execution
7. main thread check that page is loaded and fails with assertion -
because it's not loaded yet
8. IO thread executed the callback and finished the completion
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2922
- otherwise, in multi-threading environment, other thread can think that completion is finished
and start execution
- this can lead to violated assertions (for example, page must be loaded, but as callback is not executed yet
assert will be fired)
`commit_txn` in MVCC was hacking its way through I/O until now. After
adding this and the test for concurrent writers we now see `busy` errors
returning as expected because there is no `commit` queueing happening
yet until next PR I open.
Closes#2895
When we create an ImmutableRow::from_value(), we are always adding a
null padding at the end. We didn't notice this before, because a SQLite
file with an extra column is as valid as any. But that column of course
should not be there.
I traced this to column_count(), which is off by one. My understanding
is that we should be returning based on serial_types, not offset.
Closes#2862
Add expression support for `LIMIT` and `OFFSET` by storing them as
`Expr` instead of fixed integers. Constant expressions are folded with
`try_fold_to_i64`, while dynamic ones emit runtime checks, including the
new `IfNeg` opcode to clamp negative or `NULL` values to zero. The
current `build_limit_offset_expr` implementation is still naive and will
be refined in future work.
Fixes#2913Closes#2720
Now supported:
- AEGIS variants: 256, 256X2, 256X4, 128L, 128X2, 128X4
- AES-GCM variants: AES-128-GCM, AES-256-GCM
With minor changes in order to make it easy to add new ciphers later
regardless of their key size.
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#2899
This PR unifies the logic for resolving aggregate functions. Previously,
bare aggregates (e.g. `SELECT max(a) FROM t1`) and aggregates wrapped in
expressions (e.g. `SELECT max(a) + 1 FROM t1`) were handled differently,
which led to duplicated code. Now both cases are resolved consistently.
The added benchmark shows a small improvement:
```
Prepare `SELECT first_name, last_name, state, city, age + 10, LENGTH(email), UPPER(first_name), LOWE...
time: [59.791 µs 59.898 µs 60.006 µs]
change: [-7.7090% -7.2760% -6.8242%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
8 (8.00%) high mild
2 (2.00%) high severe
```
For an existing benchmark, no change:
```
Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou...
time: [11.895 µs 11.913 µs 11.931 µs]
change: [-0.2545% +0.2426% +0.6960%] (p = 0.34 > 0.05)
No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
1 (1.00%) low severe
2 (2.00%) high mild
5 (5.00%) high severe
```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2884
Now supported:
- AEGIS variants: 256, 256X2, 256X4, 128L, 128X2, 128X4
- AES-GCM variants: AES-128-GCM, AES-256-GCM
With minor changes in order to make it easy to add new
ciphers later regardless of their key size.
@penberg this PR try to clean up `turso_parser`'s`fmt` code.
- `get_table_name` and `get_column_name` should return None when
table/column does not exist.
```rust
/// Context to be used in ToSqlString
pub trait ToSqlContext {
/// Given an id, get the table name
/// First Option indicates whether the table exists
///
/// Currently not considering aliases
fn get_table_name(&self, _id: TableInternalId) -> Option<&str> {
None
}
/// Given a table id and a column index, get the column name
/// First Option indicates whether the column exists
/// Second Option indicates whether the column has a name
fn get_column_name(&self, _table_id: TableInternalId, _col_idx: usize) -> Option<Option<&str>> {
None
}
// help function to handle missing table/column names
fn get_table_and_column_names(
&self,
table_id: TableInternalId,
col_idx: usize,
) -> (String, String) {
let table_name = self
.get_table_name(table_id)
.map(|s| s.to_owned())
.unwrap_or_else(|| format!("t{}", table_id.0));
let column_name = self
.get_column_name(table_id, col_idx)
.map(|opt| {
opt.map(|s| s.to_owned())
.unwrap_or_else(|| format!("c{col_idx}"))
})
.unwrap_or_else(|| format!("c{col_idx}"));
(table_name, column_name)
}
}
```
- remove `FmtTokenStream` because it is same as `WriteTokenStream `
- remove useless functions and simplify `ToTokens`
```rust
/// Generate token(s) from AST node
/// Also implements Display to make sure devs won't forget Display
pub trait ToTokens: Display {
/// Send token(s) to the specified stream with context
fn to_tokens<S: TokenStream + ?Sized, C: ToSqlContext>(
&self,
s: &mut S,
context: &C,
) -> Result<(), S::Error>;
// Return displayer representation with context
fn displayer<'a, 'b, C: ToSqlContext>(&'b self, ctx: &'a C) -> SqlDisplayer<'a, 'b, C, Self>
where
Self: Sized,
{
SqlDisplayer::new(ctx, self)
}
}
```
Closes#2748
Currently we have `Pager::update_dirty_loaded_page_in_cache` which does
exactly what you would expect, but `DumbLruPageCache::_insert` method
with `ignore_existing` set to true, totally ignores the previous entry
and leaks the memory.
I really want to get #2885 finished and through because of the perf, but
I ran into this when inspecting it for correctness changes
Closes#2892
Revives the `MemorySim` PR and fixes a page cache issue where we could
have a unlocked and unloaded page in the page cache after a FaultyQuery.
The page would continue in the cache and could affect other queries as
the `page_cache` is at the `Connection` level.
Depends on #2785Closes#2693
This PR updates the internal encryption framework to handle
authentication tags explicitly rather than relying on the underlying
cipher libraries to append/verify them automatically.
closes: #2850
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#2858
This also gave a small performance boost.
Local run results:
```
Prepare `SELECT first_name, last_name, state, city, age + 10, LENGTH(email), UPPER(first_name), LOWE...
time: [59.791 µs 59.898 µs 60.006 µs]
change: [-7.7090% -7.2760% -6.8242%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
8 (8.00%) high mild
2 (2.00%) high severe
```
Aggregate functions cannot be nested, and this is validated during the
translation of aggregate function arguments. Therefore, traversing their
child expressions is unnecessary.
Handled in the same way as in `prepare_one_select_plan` for bare
function calls. In `prepare_one_select_plan`, however, resolving
external scalar functions is performed unnecessarily twice.
The new query combines multiple aggregate functions, plain columns,
arithmetic expressions, and aggregates wrapped in additional expressions.
Local run results:
```
Prepare `SELECT first_name, last_name, state, city, age + 10, LENGTH(email), UPPER(first_name), LOWE...
time: [64.535 µs 64.623 µs 64.713 µs]
Found 9 outliers among 100 measurements (9.00%)
4 (4.00%) high mild
5 (5.00%) high severe
```
The initial commits fix issues and plug gaps between ungrouped and
grouped aggregations.
The final commit consolidates the code that emits `AggStep` to prevent
future disparities between the two.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2867
Some progress working on `printf` support.
(relevant issue https://github.com/tursodatabase/turso/issues/885)
Implementation of the basic substitution types cited in the `TODO`
comment on the beginning of the file (%i, %x, %X, %o, %e, %E, %c). There
are some others in the sqlite spec which I will implement in a future
PR.
I tried to pay attention to the specific behaviors from sqlite as much
as possible while testing this, but if there's something I missed please
tell me.
Also, I see this code needs to be reorganized already, I'm still
thinking on the best approach to do that without affecting the
ergonomics of new implementations, I'm still learning Rust so this is
not obvious for me right now. I'm open to suggestions about it.
Closes#2868
Because we can abort a read_page completion, this means a page can be in
the cache but be unloaded and unlocked. However, if we do not evict that
page from the page cache, we will return an unloaded page later which
will trigger assertions later on. This is worsened by the fact that page
cache is not per `Statement`, so you can abort a completion in one
Statement, and trigger some error in the next one if we don't evict the
page in these circumstances.
Also, to propagate IO errors we need to return the Error from
IOCompletions on step.
Closes#2785
Using `usize` to compute file offsets caps us at ~16GB on 32-bit
systems. For example, with 4 KiB pages we can only address up to 1048576
pages; attempting the next page overflows a 32-bit usize and can wrap
the write offset, corrupting data. Switching our I/O APIs and offset
math to u64 avoids this overflow on 32-bit targets
Closes#2791
