Commit Graph

9034 Commits

Author SHA1 Message Date
Pekka Enberg
84cf3640cb Merge 'Enable encryption option in Whopper ' from Avinash Sajjanshetty
If encryption is arg is passed, then we run whopper tests with
encryptions. We randomly generate cipher and key, and run whopper. They
are also printed for debugging and analysis.
Also, updated the corresponding scripts. So now you can do:
```bash
./whopper/bin/run --enable-encryption

or 

./whopper/bin/explore --enable-encryption
```

Closes #3183
2025-09-18 10:03:13 +03:00
Jussi Saurio
1d2b461a2a Merge 'Compat: Translate the 2nd argument of group_concat / string_agg' from Iaroslav Zeigerman
Fixes #3140

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3155
2025-09-18 09:23:05 +03:00
Jussi Saurio
0d37ac2519 Merge 'translate: couple fixes from testing with Gorm' from Preston Thorpe
Ongoing tests for [turso-go](https://github.com/tursodatabase/turso-go)
have unearthed a couple more issues
closes #3187
### Number 1:
We were getting something like:
```sql
sqlite_autoindex_`databases`_2
```
when creating autoindex for table in Gorm (gorm is notorious for
backticks everywhere), because of not normalizing the column name when
creating autoindex.
### Number 2:
When creating table with `PRIMARY KEY AUTOINCREMENT`, we were still
creating the index, but it wasn't properly handled in
`populate_indices`, because we are doing the following:
```rust
                if column.primary_key && unique_set.is_primary_key {
                    if column.is_rowid_alias {
                        // rowid alias, no index needed
                        continue; // continues, but doesn't consume it..
                    }
```
So if we created such an index entry for the AUTOINCREMENT... we would
trip this:
```rust
assert!(automatic_indexes.is_empty(), "all automatic indexes parsed from sqlite_schema should have been consumed, but {} remain", automatic_indexes.len());
```

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3186
2025-09-18 09:21:41 +03:00
Jussi Saurio
498293658e Merge 'Reduce allocations needed for break_predicate_at_and_boundaries' from Lâm Hoàng Phúc
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3188
2025-09-18 09:21:24 +03:00
Jussi Saurio
91ef4e5e9d Merge 'Introduce instruction VTABLE' from Lâm Hoàng Phúc
this PR improves 3-6% for `prepare` benchmark without slowing down
others.  After this PR we don't have to store `InsnFunction`  in
`Program` and `ProgramBuilder` anymore, because `to_function` will
return result without matching.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #3098
2025-09-18 09:18:48 +03:00
Jussi Saurio
26f56d794e Merge 'Simulator Multiple Connections' from Pedro Muniz
This is the start of the work to add transactional properties to the
Simulator. Before this PR, we only executed the queries on one
connection, now the default is to run with 10 connections. Each
connection is selected at random to run the database queries.
A lot of the code was hardcoded to work with 1 connection only, so most
of the refactor came from me adjusting the `Interaction` and
`Interactions` structs to hold a connection index. As I had to refactor
a lot of code, I took the opportunity to deduplicate code that across
`run_simulation_default`, `watch::run_simulation`,
`differential::run_simulation`, `doublecheck::run_simulation`.
In my next PR, I intend to add transaction isolation checking which will
involve some more refactoring to ensure the simulator correctly tracks
transaction state and table visibility.
And in a future PR, we can extend this further with concurrent
transactions as well

Closes #3163
2025-09-18 09:10:01 +03:00
Pekka Enberg
d2376a239a Merge 'core/mvcc: introduce with_header for MVCC header update tracking' from Pere Diaz Bou
Currently header changes are tracked through pager by reading page 1.
MVCC has it's own layer to track changes during txn so this commit makes
it so that headers are tracked by each txn separately.
On commit we update the _global_ header which is used to update
`database_size` because pager commits require it to be up to date. This
also makes it _simpler_ to keep track of header updates and update
pager's header accordingly.
This PR is needed in order to make logical log work because we don't
want to rely on pager as much as possible!

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3156
2025-09-18 08:13:14 +03:00
TcMits
88119888d0 reduce allocation needed for break_predicate_at_and_boundaries 2025-09-18 10:52:29 +07:00
PThorpe92
5aa07eb826 Use normalized table name for autoindex creation 2025-09-17 20:51:22 -04:00
PThorpe92
45fcadbf20 Fix formatting of autoindex 2025-09-17 16:07:11 -04:00
PThorpe92
faa3113c77 Add regression test for table creation pk autoincrement constraint with backticks 2025-09-17 15:53:35 -04:00
PThorpe92
cfc6ec4ee5 Normalize more idents 2025-09-17 15:53:06 -04:00
PThorpe92
c57545d504 Avoid panicking when we create autoindex for AUTOINCREMENT primary key 2025-09-17 15:52:42 -04:00
PThorpe92
dde8a49f4e normalize identifier for creating autoindex to prevent e.g. sqlite_autoindextable_2 2025-09-17 13:25:33 -04:00
PThorpe92
4e71524e42 normalize identifier for ID::Name in upsert expr rewriting 2025-09-17 13:24:06 -04:00
Avinash Sajjanshetty
94e7f6defc Update whopper run and explore scripts 2025-09-17 21:58:54 +05:30
Avinash Sajjanshetty
10137ffaa4 run whopper with encryption if arg is passed 2025-09-17 21:58:27 +05:30
Avinash Sajjanshetty
f9a6cde79a Make encryption::CipherMode public 2025-09-17 21:55:23 +05:30
Preston Thorpe
8c53d7f024 Merge 'translation: rewrite expressions and properly handle quoted identifiers in UPSERT' from Preston Thorpe
This PR fixes bugs found in the [turso-
go](https://github.com/tursodatabase/turso-go) driver with UPSERT clause
earlier, where `Gorm` will (obviously) use Expr::Variable's as well as
use quotes for `Expr::Qualified` in the tail end of an UPSERT statement.
Example:
```sql
INSERT INTO users (a,b,c) VALUES (?,?,?) ON CONFLICT (`users`.`a`) DO UPDATE SET b = `excluded`.`b`, a = ?;
```
and previously we were not properly calling `rewrite_expr`, which was
not properly setting the anonymous `Expr::Variable` to `__param_N` named
parameter, so it would ignore it completely, then return the wrong # of
parameters.
Also, we didn't handle quoted "`excluded`.`x`", so it would panic in the
optimizer that Qualified should have been rewritten earlier.

Closes #3157
2025-09-17 11:25:13 -04:00
Iaroslav Zeigerman
29e0cabf2a Compat: Translate the 2nd argument of group_concat / string_agg 2025-09-17 07:42:07 -07:00
Jussi Saurio
8bf52de94b Merge 'Remove serialization of normal write/commit path' from Preston Thorpe
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3089
2025-09-17 17:30:45 +03:00
pedrocarlo
27a8f019c6 clippy 2025-09-17 10:49:24 -03:00
pedrocarlo
8b05dcba21 add 10 connections to base profiles 2025-09-17 10:49:24 -03:00
pedrocarlo
6039a30222 fix double create failure 2025-09-17 10:49:24 -03:00
pedrocarlo
81cff3e3d7 fix shrinking to use a secondary index that points to the Property that generated the interactions 2025-09-17 10:49:24 -03:00
pedrocarlo
04154bf368 simplify code for doublecheck testing 2025-09-17 10:49:24 -03:00
pedrocarlo
f95ac87f06 simplify code for differential testing 2025-09-17 10:49:24 -03:00
pedrocarlo
4d9e676f65 simplify run_simualtion signature + remove watch mode file as it shares logic with default run simulation logic 2025-09-17 10:49:24 -03:00
pedrocarlo
35ddcb4270 adjust shrinking as we do not have a property pointer 2025-09-17 10:49:24 -03:00
pedrocarlo
b98c22a90e run simulation on the total list of interactions to avoid recomputing whole vec of interactions on each loop + change state management in preparation for multiple connections change 2025-09-17 10:49:24 -03:00
pedrocarlo
faf38fe196 add connection Index to interaction struct 2025-09-17 10:49:24 -03:00
pedrocarlo
7e02fbe7e7 make assertion to be cloneable 2025-09-17 10:49:24 -03:00
pedrocarlo
77cab30c5b add connection index to interaction 2025-09-17 10:49:24 -03:00
Pekka Enberg
f9db3e97fd Merge 'core/vtab: Wrap InternalVirtualTable with RwLock' from Pekka Enberg
Closes #3180
2025-09-17 16:40:20 +03:00
Pekka Enberg
617adf8eba Merge 'Clean up encryption feature flag usage' from Avinash Sajjanshetty
Closes #3178
2025-09-17 16:40:08 +03:00
Pekka Enberg
5ebf6cd128 core/vtab: Wrap InternalVirtualTable with RwLock 2025-09-17 15:43:33 +03:00
Pekka Enberg
b98e1efb17 Merge 'core/storage: Wrap Pager::checkpoint_state in RwLock' from Pekka Enberg
Closes #3176
2025-09-17 14:41:18 +03:00
Avinash Sajjanshetty
4eaee0fd81 clean up encryption feature flag usage 2025-09-17 16:30:51 +05:30
Preston Thorpe
bcafb288ad Merge 'Fix is_nonnull returns true on 1 / 0' from Lâm Hoàng Phúc
turso:
```sh
turso> CREATE TABLE t (x PRIMARY KEY, y, z);
turso> INSERT INTO t VALUES (37, -70, -196792117);
turso> SELECT * FROM t WHERE  (1 / 0) >= -3289742039 < t.x;
┌────┬─────┬────────────┐
│ x  │ y   │ z          │
├────┼─────┼────────────┤
│ 37 │ -70 │ -196792117 │
└────┴─────┴────────────┘
turso>
```
sqlite:
```sh
sqlite> CREATE TABLE t (x PRIMARY KEY, y, z);
sqlite> INSERT INTO t VALUES (37, -70, -196792117);
sqlite> SELECT * FROM t WHERE  (1 / 0) >= -3289742039 < t.x;
sqlite>
```
related: https://github.com/tursodatabase/turso/actions/runs/17765571409
/job/50488042583?pr=3147#step:8:855

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #3167
2025-09-17 06:55:55 -04:00
PThorpe92
97c11898fe Minor refactor in translate/insert 2025-09-17 06:44:10 -04:00
PThorpe92
8dd1cf11eb Add some more specific tests for upsert 2025-09-17 06:44:10 -04:00
PThorpe92
5dd466941e Handle upsert even in inserting_multiple_rows case 2025-09-17 06:44:09 -04:00
PThorpe92
85eee42bf1 Support quoted qualified identifiers in UPSERT excluded.x clauses 2025-09-17 06:44:08 -04:00
PThorpe92
d2cd833b86 Rewrite exprs in set + where clause for UPSERT 2025-09-17 06:38:25 -04:00
Pekka Enberg
2b89dd4ea8 Merge 'core: Wrap Pager dirty_pages in RwLock' from Pekka Enberg
Make it Sync and Send.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3175
2025-09-17 13:34:47 +03:00
Pekka Enberg
58b0e8665d Merge 'core: Wrap MvCursor in Arc<RwLock<>>' from Pekka Enberg
Make it Send and Sync.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3174
2025-09-17 13:34:34 +03:00
Pere Diaz Bou
434daf4b10 core/mvcc: fmt 2025-09-17 12:18:44 +02:00
Pere Diaz Bou
1627590bc3 core/mvcc: add expect panic in case txn was not found 2025-09-17 12:12:24 +02:00
Pekka Enberg
cb68c953e1 Merge 'core/incremental: Wrap ViewTransactionState in Arc' from Pekka Enberg
Make it Send.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3173
2025-09-17 13:07:54 +03:00
Pekka Enberg
a2f6ff819a Merge 'core/function: Wrap ExtFunc in Arc' from Pekka Enberg
Make it Send.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3172
2025-09-17 13:07:43 +03:00