Commit Graph

10673 Commits

Author SHA1 Message Date
PThorpe92
30454336a6 Make io_uring sound for connections across multiple threads 2025-09-03 10:54:42 -04:00
Nikita Sivukhin
9f282cf1e0 update readme 2025-09-03 18:21:22 +04:00
Nikita Sivukhin
62e7384771 build native package for ARM64 2025-09-03 17:57:15 +04:00
Nikita Sivukhin
185392fb9d skip optional packages publish as they will be published for native package 2025-09-03 17:26:36 +04:00
Nikita Sivukhin
40dd9be201 introduce package.json for separate *-browser package (both database and sync) 2025-09-03 17:15:47 +04:00
Pere Diaz Bou
8db5cead07 core/mvcc: only commit if there is a txn 2025-09-03 14:12:48 +02:00
Pere Diaz Bou
b8f83e1fc0 clippy and fmt stuff because if not pekka will tweet 2025-09-03 12:47:55 +02:00
sonhmai
2b6cb39c7e core: handle edge cases for read_varint 2025-09-03 15:43:34 +07:00
TcMits
b6fca2718f fmt 2025-09-03 13:41:23 +07:00
TcMits
b0f4dd49d5 use match_ignore_ascii_case macro 2025-09-03 12:01:52 +07:00
Frank Denis
52d0a3bf4a Make set_encryption_{context,cipher,key} fallible 2025-09-03 01:14:49 +02:00
Frank Denis
e3835afee5 Encryption: add support for other AEGIS and AES-GCM cipher variants
Now supported:

- AEGIS variants: 256, 256X2, 256X4, 128L, 128X2, 128X4
- AES-GCM variants: AES-128-GCM, AES-256-GCM

With minor changes in order to make it easy to add new
ciphers later regardless of their key size.
2025-09-02 23:46:58 +02:00
themixednuts
fa19ba4993 chore: add tests 2025-09-02 10:58:00 -05:00
Pekka Enberg
1de647758f Merge 'refactor parser fmt' from Lâm Hoàng Phúc
@penberg this PR try to clean up `turso_parser`'s`fmt` code.
- `get_table_name` and `get_column_name` should return None when
table/column does not exist.
```rust
/// Context to be used in ToSqlString
pub trait ToSqlContext {
    /// Given an id, get the table name
    /// First Option indicates whether the table exists
    ///
    /// Currently not considering aliases
    fn get_table_name(&self, _id: TableInternalId) -> Option<&str> {
        None
    }

    /// Given a table id and a column index, get the column name
    /// First Option indicates whether the column exists
    /// Second Option indicates whether the column has a name
    fn get_column_name(&self, _table_id: TableInternalId, _col_idx: usize) -> Option<Option<&str>> {
        None
    }

    // help function to handle missing table/column names
    fn get_table_and_column_names(
        &self,
        table_id: TableInternalId,
        col_idx: usize,
    ) -> (String, String) {
        let table_name = self
            .get_table_name(table_id)
            .map(|s| s.to_owned())
            .unwrap_or_else(|| format!("t{}", table_id.0));

        let column_name = self
            .get_column_name(table_id, col_idx)
            .map(|opt| {
                opt.map(|s| s.to_owned())
                    .unwrap_or_else(|| format!("c{col_idx}"))
            })
            .unwrap_or_else(|| format!("c{col_idx}"));

        (table_name, column_name)
    }
}
```
- remove `FmtTokenStream` because it is same as `WriteTokenStream `
- remove useless functions and simplify `ToTokens`
```rust
/// Generate token(s) from AST node
/// Also implements Display to make sure devs won't forget Display
pub trait ToTokens: Display {
    /// Send token(s) to the specified stream with context
    fn to_tokens<S: TokenStream + ?Sized, C: ToSqlContext>(
        &self,
        s: &mut S,
        context: &C,
    ) -> Result<(), S::Error>;

    // Return displayer representation with context
    fn displayer<'a, 'b, C: ToSqlContext>(&'b self, ctx: &'a C) -> SqlDisplayer<'a, 'b, C, Self>
    where
        Self: Sized,
    {
        SqlDisplayer::new(ctx, self)
    }
}
```

Closes #2748
2025-09-02 18:35:43 +03:00
Pekka Enberg
2addeb5a9f Merge 'introduce eq/contains/starts_with/ends_with_ignore_ascii_case macros' from Lâm Hoàng Phúc
depend on #2865
```sh
`ALTER TABLE _ RENAME TO _`/limbo_rename_table/
                        time:   [10.100 ms 10.191 ms 10.283 ms]
                        change: [-16.770% -15.559% -14.417%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild

`ALTER TABLE _ RENAME COLUMN _ TO _`/limbo_rename_column/
                        time:   [7.4829 ms 7.5492 ms 7.6128 ms]
                        change: [-19.397% -18.093% -16.789%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) low mild
  1 (1.00%) high mild

`ALTER TABLE _ ADD COLUMN _`/limbo_add_column/
                        time:   [5.3255 ms 5.3713 ms 5.4183 ms]
                        change: [-24.002% -22.612% -21.195%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 39 outliers among 100 measurements (39.00%)
  17 (17.00%) low severe
  1 (1.00%) low mild
  1 (1.00%) high mild
  20 (20.00%) high severe

`ALTER TABLE _ DROP COLUMN _`/limbo_drop_column/
                        time:   [5.8858 ms 5.9183 ms 5.9510 ms]
                        change: [-16.233% -14.679% -13.083%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 25 outliers among 100 measurements (25.00%)
  8 (8.00%) low severe
  11 (11.00%) low mild
  2 (2.00%) high mild
  4 (4.00%) high severe

Prepare `SELECT 1`/limbo_parse_query/SELECT 1
                        time:   [590.28 ns 591.31 ns 592.35 ns]
                        change: [-3.7810% -3.5059% -3.2444%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low severe
  6 (6.00%) high mild

Prepare `SELECT * FROM users LIMIT 1`/limbo_parse_query/SELECT * FROM users LIMIT 1
                        time:   [1.2569 µs 1.2582 µs 1.2596 µs]
                        change: [-5.0125% -4.7516% -4.4933%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

Prepare `SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1) > 1 ORDER BY cou...
                        time:   [3.7180 µs 3.7227 µs 3.7274 µs]
                        change: [-3.0557% -2.7642% -2.4761%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild

Execute `SELECT 1`/limbo_execute_select_1
                        time:   [27.455 ns 27.477 ns 27.499 ns]
                        change: [-2.9461% -2.7493% -2.5589%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

Execute `SELECT * FROM users LIMIT ?`/limbo_execute_select_rows/1
                        time:   [410.53 ns 411.05 ns 411.54 ns]
                        change: [-15.364% -15.133% -14.912%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) low mild
  1 (1.00%) high mild

Execute `SELECT * FROM users LIMIT ?`/limbo_execute_select_rows/10
                        time:   [2.1100 µs 2.1122 µs 2.1145 µs]
                        change: [-11.517% -11.065% -10.662%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) low severe
  2 (2.00%) low mild

Execute `SELECT * FROM users LIMIT ?`/limbo_execute_select_rows/50
                        time:   [9.5156 µs 9.5268 µs 9.5383 µs]
                        change: [-10.284% -10.086% -9.8833%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild

Execute `SELECT * FROM users LIMIT ?`/limbo_execute_select_rows/100
                        time:   [18.669 µs 18.698 µs 18.731 µs]
                        change: [-9.5949% -9.3407% -9.1140%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low severe
  1 (1.00%) high mild

Execute `SELECT count() FROM users`/limbo_execute_select_count
                        time:   [7.1027 µs 7.1098 µs 7.1170 µs]
                        change: [-43.739% -43.596% -43.469%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe

```

Closes #2866
2025-09-02 18:35:14 +03:00
Pekka Enberg
adc6cb008a Merge 'introduce match_ignore_ascii_case macro' from Lâm Hoàng Phúc
this PR converts generate code in `turso_parser`'s `build.rs` into macro
for reusability. `match_ignore_ascii_case` will generate trie-like tree
matching from normal match expression.
example:
```rust
    match_ignore_ascii_case!(match input {
        b"AB" => TokenType::TK_ABORT,
        b"AC" => TokenType::TK_ACTION,
        _ => TokenType::TK_ID,
    })
```
will generate:
```rust
    match input.get(0) {
        Some(b'A') | Some(b'a') => match input.get(1) {
            Some(b'B') | Some(b'b') => match input.get(2) {
                None => TokenType::TK_ABORT,
                _ => TokenType::TK_ID,
            },
            Some(b'C') | Some(b'c') => match input.get(2) {
                None => TokenType::TK_ACTION,
                _ => TokenType::TK_ID,
            },
            _ => TokenType::TK_ID,
        },
        _ => TokenType::TK_ID,
    }
```

Closes #2865
2025-09-02 18:34:55 +03:00
Pekka Enberg
d77b76e75a Merge 'string sometimes used as identifier quoting' from Lâm Hoàng Phúc
fix https://github.com/tursodatabase/turso/issues/2886#issuecomment-
3244885481

Closes #2894
2025-09-02 18:34:43 +03:00
Pekka Enberg
36909d4551 Merge 'CURRENT_TIMESTAMP can fallback TK_ID' from Lâm Hoàng Phúc
fix #2875

Closes #2893
2025-09-02 18:34:19 +03:00
Pekka Enberg
52ef7dd675 Merge 'Fix memory leak in page cache during balancing' from Preston Thorpe
Currently we have `Pager::update_dirty_loaded_page_in_cache` which does
exactly what you would expect, but `DumbLruPageCache::_insert` method
with `ignore_existing` set to true, totally ignores the previous entry
and leaks the memory.
I really want to get #2885 finished and through because of the perf, but
I ran into this when inspecting it for correctness changes

Closes #2892
2025-09-02 18:32:56 +03:00
Pekka Enberg
4e5274e646 Merge 'core: Make strict schema support experimental' from Pekka Enberg
It's not tested properly so let's mark it as experimental for now.
Fixes #2775

Closes #2891
2025-09-02 18:31:31 +03:00
Pere Diaz Bou
13c505109a core/mvcc: make commit_txn return on I/O 2025-09-02 17:07:38 +02:00
Pere Diaz Bou
f26e36b6de core/mvcc: test write concurrency
fix
2025-09-02 17:07:38 +02:00
TcMits
635402fc6f string sometimes used as identifier quoting 2025-09-02 21:35:37 +07:00
PThorpe92
cfadc4f579 Fix memory leak in page cache during balancing 2025-09-02 10:35:04 -04:00
TcMits
d0cb3d0d08 CURRENT_TIMESTAMP can fallback TK_ID 2025-09-02 20:50:58 +07:00
Pekka Enberg
12cf4d2e72 core: Make strict schema support experimental
It's not tested properly so let's mark it as experimental for now.

Fixes #2775
2025-09-02 16:40:02 +03:00
TcMits
40adf3fcfd Merge branch 'perf-3' into perf-4 2025-09-02 18:47:05 +07:00
TcMits
53d239ad1e resolve conflict 2025-09-02 18:46:41 +07:00
TcMits
bc8b848aba docs 2025-09-02 18:43:12 +07:00
TcMits
bfff05faba merge main 2025-09-02 18:25:20 +07:00
Pekka Enberg
8f7e43b32b scripts/publish-crates.sh: Remove turso_sqlite3_parser package 2025-09-02 14:21:04 +03:00
Pekka Enberg
3f3125c4f4 Merge 'remove turso_sqlite3_parser from turso_parser' from Lâm Hoàng Phúc
Closes #2888
2025-09-02 14:20:39 +03:00
TcMits
06e14c8ace merge main 2025-09-02 18:17:37 +07:00
Pekka Enberg
8204fbc8ec simulator: Fix 64-bit offset build failures
Fix brekage from first merging commit d959319b ("Merge 'Use u64 for file
offsets in I/O and calculate such offsets in u64' from Preston Thorpe")
and then commit 6591b66c ("Merge 'Simulate I/O in memory' from Pedro
Muniz"), which was unaware of the changes.
2025-09-02 14:14:04 +03:00
TcMits
d298480e4a Merge branch 'main' into perf-3 2025-09-02 18:13:58 +07:00
TcMits
07feacbc76 remove turso_sqlite3_parser from turso_parser 2025-09-02 18:10:28 +07:00
TcMits
33a04fbaf7 resolve conflict 2025-09-02 17:30:10 +07:00
Pekka Enberg
6591b66c3d Merge 'Simulate I/O in memory' from Pedro Muniz
Revives the `MemorySim` PR and fixes a page cache issue where we could
have a unlocked and unloaded page in the page cache after a FaultyQuery.
The page would continue in the cache and could affect other queries as
the `page_cache` is at the `Connection` level.
Depends on #2785

Closes #2693
2025-09-02 13:28:48 +03:00
Pekka Enberg
3ec6f37555 scripts: Add turso_sqlite3_parser back to publish-crates.sh
Turns out turso_parser depends on it.
2025-09-02 12:52:40 +03:00
Pekka Enberg
91a51c170e Fix up turso_parser version in Cargo.toml 2025-09-02 12:47:49 +03:00
Pekka Enberg
6c7936a016 Turso 0.1.5-pre.2 2025-09-02 12:45:41 +03:00
Pekka Enberg
adaf0bec6c scripts: Fix publish-crates.sh to publish new parser crate 2025-09-02 12:45:16 +03:00
Pekka Enberg
483cd92dab Drop removed Go bindings from Antithesis Dockerfile 2025-09-02 09:48:53 +03:00
Pekka Enberg
15d45e3f68 Merge 'Refactor encryption to manage authentication tag internally' from bit-aloo
This PR updates the internal encryption framework to handle
authentication tags explicitly rather than relying on the underlying
cipher libraries to append/verify them automatically.
closes: #2850

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2858
2025-09-02 09:44:22 +03:00
Piotr Rzysko
e97cc64ad0 Remove duplicated code for resolving aggregates
This also gave a small performance boost.

Local run results:

```
Prepare `SELECT first_name, last_name, state, city, age + 10, LENGTH(email), UPPER(first_name), LOWE...
                        time:   [59.791 µs 59.898 µs 60.006 µs]
                        change: [-7.7090% -7.2760% -6.8242%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  8 (8.00%) high mild
  2 (2.00%) high severe
```
2025-09-02 08:22:37 +02:00
Piotr Rzysko
517f23013a Delay deduplication of aggregate expressions
It is not necessary to iterate over existing aggregates for every
traversed expression. Instead, do so only when an aggregate function
is found.
2025-09-02 08:22:37 +02:00
Piotr Rzysko
569e41cb1e Skip traversing children of aggregate functions
Aggregate functions cannot be nested, and this is validated during the
translation of aggregate function arguments. Therefore, traversing their
child expressions is unnecessary.
2025-09-02 08:22:37 +02:00
Piotr Rzysko
9b742a64c2 Handle functions with star argument wrapped in expressions
Handled in the same way as in `prepare_one_select_plan` for bare
function calls.
2025-09-02 08:22:36 +02:00
Piotr Rzysko
f3cbc382ce Support external aggregate functions wrapped in expressions
Handled in the same way as in `prepare_one_select_plan` for bare
function calls. In `prepare_one_select_plan`, however, resolving
external scalar functions is performed unnecessarily twice.
2025-09-02 08:22:36 +02:00
Piotr Rzysko
d361734819 Remove unnecessary recursion in resolve_aggregates
The walk_expr method already traverses arguments, so there is no need
to do this explicitly.
2025-09-02 08:22:36 +02:00