Commit Graph

10673 Commits

Author SHA1 Message Date
PThorpe92
7282ed38b1 Remove serialization of normal write/commit path 2025-09-14 09:17:38 -04:00
TcMits
0e2c043535 unrelated changes 2025-09-14 19:04:41 +07:00
TcMits
4bb6b02b65 clean PR 2025-09-14 19:03:09 +07:00
TcMits
a658273c63 fmt 2025-09-14 18:59:57 +07:00
TcMits
cab0c7b545 peft tuning 2025-09-14 18:53:53 +07:00
Pekka Enberg
1bdfabcac7 whopper: Generate different transaction modes with MVCC 2025-09-14 13:55:08 +03:00
Pekka Enberg
5783e3edf8 whopper: Add "--enable-mvcc" command line option 2025-09-14 13:15:52 +03:00
Pekka Enberg
db190c9e3d whopper: Add "--disable-indexes" option
MVCC does not support indexes so let's add an option to disable that.
2025-09-14 13:13:49 +03:00
Pekka Enberg
95660535da core/storage: Demote info logging to debug 2025-09-14 13:10:46 +03:00
Pekka Enberg
a5f1bdbc9d Merge 'perf/throughput/turso: Don't interleave concurrent transactions in on…' from Pekka Enberg
…e connection
With BEGIN CONCURRENT, we cannot use the same connection all the time
because there's no way for the transaction manager to know which
transaction belongs to what "session" -- they're all individual
statements executed in the context of the one connection.
Fixes #3093

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3094
2025-09-14 11:33:20 +03:00
Pekka Enberg
ed8b02d83e perf/throughput/turso: Don't use spawn_blocking()
Let's just use normal spawn()
2025-09-14 11:32:17 +03:00
Pekka Enberg
222b9e7ca2 Merge 'perf/throughput/turso: Don't execute futures serially' from Pekka Enberg
Looping through the futures means we're essentially executing them
serially.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3092
2025-09-14 11:01:06 +03:00
Pekka Enberg
6e9ba4c561 perf/throughput/turso: Don't interleave concurrent transactions in one connection
With BEGIN CONCURRENT, we cannot use the same connection all the time
because there's no way for the transaction manager to know which
transaction belongs to what "session" -- they're all individual
statements executed in the context of the one connection.

Fixes #3093
2025-09-14 10:52:56 +03:00
Pekka Enberg
98af9386e2 perf/throughput/turso: Don't execute futures serially
Looping through the futures means we're essentially executing them
serially.
2025-09-14 10:07:21 +03:00
Pavan-Nambi
037c3892bb MemMax impl 2025-09-14 09:10:20 +05:30
Preston Thorpe
4e4f0ea61f Merge 'Dont grab page cache write lock in a loop' from Preston Thorpe
durp

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3088
2025-09-13 12:48:19 -04:00
PThorpe92
f6dd0bc4d6 Dont grab page cache write lock in a loop 2025-09-13 12:21:13 -04:00
Pavan-Nambi
255cfb10e6 merge autoincrement into translate insert 2025-09-13 21:21:28 +05:30
Pekka Enberg
6a2f0d6061 Merge 'Add per page checksums' from Avinash Sajjanshetty
This patch adds checksums to Turso DB. You may check the design here in
the [RFC](https://github.com/tursodatabase/turso/issues/2178).
1. We use reserved bytes (8 bytes) to store the checksums. On every IO
read, we verify that the checksum matches.
2. We use twox hash for checksums.
3. Checksum works only on 4K pages now. It's a small change to enable
for all other sizes, I will send another PR.
4. Right now, it's not possible to switch to different algorithm or turn
off altogether. That will be added in the future PRs.
5. Checksums can be enabled only for new dbs. For existing DBs, we will
disable it.
6. To add checksums for existing DBs, we need vacuum since it would
require rewrite of whole db.

Closes #2840
2025-09-13 18:46:53 +03:00
Pekka Enberg
7f5038f7c9 Merge 'perf/throughput/turso: Async transactions with concurrent mode' from Pekka Enberg
With `BEGIN CONCURRENT`, we should also take advantage of async
transaction processing to maximize concurrency.

Closes #3082
2025-09-13 15:07:29 +03:00
TcMits
e18b6b0b56 inline 2025-09-13 18:07:45 +07:00
Pekka Enberg
7d3ce68695 Merge 'core/throughput: Add per transaction think time support' from Pekka Enberg
Closes #3080
2025-09-13 14:07:30 +03:00
Pavan-Nambi
0effb981e6 autoincrement functionality works as good as sqlite now, handled all edge cases that we are aware of
- The code now prevents dropping or indexing `sqlite_sequence`
- make sure that AUTOINCREMENT only works on a single `INTEGER PRIMARY KEY`
-  handles `i64::MAX` gracefully by returning `SQLITE_FULL`
- also AUTOINCREMENT now works in both column and table constraints.

fmt
2025-09-13 16:35:36 +05:30
Pekka Enberg
898f32f7f7 Fix Antithesis Dockerfile to include whopper 2025-09-13 13:33:11 +03:00
Pekka Enberg
3733d3856a Merge 'core: Panic on fsync() error by default' from Pekka Enberg
Retrying fsync() on error was historically not safe ("fsyncgate") and
Postgres still defaults to panicing on fsync(). Therefore, add a
"data_sync_retry" pragma (disabled by default) and use it to determine
whether to panic on fsync() error or not.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3077
2025-09-13 13:32:07 +03:00
Pekka Enberg
0fad30a30d perf/throughput/turso: Async transactions with concurrent mode
With `BEGIN CONCURRENT`, we should also take advantage of async
transaction processing to maximize concurrency.
2025-09-13 13:25:16 +03:00
Pekka Enberg
8dc2e738a4 core/throughput: Add per transaction think time support 2025-09-13 13:02:43 +03:00
TcMits
17ee979583 fix macro 2025-09-13 16:35:58 +07:00
TcMits
01da48fde9 introduce instruction virtual table 2025-09-13 16:35:17 +07:00
Piotr Rzysko
1a95131c3c Include windows in ToTokens for SelectPlan 2025-09-13 11:12:44 +02:00
Piotr Rzysko
9ff2133ff2 Rewrite window function expressions in the optimizer
Currently, this is effectively a no-op because, at the optimization
stage, window function expressions are in the form
win_func(subquery_column1, subquery_column2, ...).

Nevertheless, expressions are rewritten to maintain consistency with
aggregates, which also hold cloned expressions from sources like result
columns. This ensures future changes in the optimizer won’t break window
function handling.
2025-09-13 11:12:44 +02:00
Piotr Rzysko
f5efcbe745 Add support for window functions
Adds initial support for window functions. For now, only existing
aggregate functions can be used as window functions—no specialized
window-specific functions are supported yet.

Currently, only the default frame definition is implemented:
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE NO OTHERS.
2025-09-13 11:12:44 +02:00
Piotr Rzysko
c81cd16230 Extract QueryDestination::placeholder_for_subquery 2025-09-13 10:49:14 +02:00
Piotr Rzysko
1826023c32 Decouple AggArgumentSource::Expression from Aggregate
This allows it to be reused for window function processing without
relying on the Aggregate struct.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
6c3c44e204 Expose fewer details from AggArgumentSource
Hides unnecessary internals to decouple the API from the Aggregate struct.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
5f2a3e1242 Handle dummy argument for count() and count(*) in translation
Two main reasons for this change:
* Improve readability by moving the logic for this special case closer
  to the code that relies on it.
* Decouple AggFunc from the Aggregate struct. In the future, window
  function processing will use AggFunc directly, without necessarily
  depending on Aggregate.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
6d84cbedc2 Fix delimiter handling in group_concat and string_agg
Non-literal delimiters must be translated by AggArgumentSource.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
110ffba2a1 Fix accumulator reset when arguments outnumber aggregates
Previously, while resetting accumulator registers, we would also
reset subsequent registers. This happened because the number of registers
to reset was computed as the sum of arguments rather than the number of
aggregate functions.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
6224cdbbd3 Support WalkControl in walk_expr_mut
Now walk_expr_mut can use WalkControl to skip parts of the expression
tree. This makes it consistent with walk_expr.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
b911e80607 Add AggValue instruction
Adds the AggValue instruction, which computes the current aggregate
result and writes it to a dedicated destination register.

Unlike AggFinal, it does not overwrite or clear the accumulator
register. This makes it possible to retrieve aggregate results multiple
times—needed when processing window functions—while preserving the
accumulator state.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
c5a12f52c2 Don't mutate state in op_agg_final
Previously, only the External and Avg aggregates mutated state during
AggFinal. This is unnecessary because AggFinal runs only once per group,
so caching the result provides no performance benefit.

By avoiding state mutation, we can also reuse op_agg_final for the
AggValue instruction that will be added soon.
2025-09-13 10:49:14 +02:00
Piotr Rzysko
458172220e Remove unused method from AggContext 2025-09-13 10:49:14 +02:00
Piotr Rzysko
867bef55d8 Add ResetSorter instruction
This instruction isn't used yet, but it will be needed for window
functions, since they heavily rely on ephemeral tables.
2025-09-13 10:44:56 +02:00
Piotr Rzysko
ea9599681e Add OpenDup instruction
The instruction isn’t used yet, but it’ll be needed for window functions,
since they heavily rely on ephemeral tables.
2025-09-13 10:35:33 +02:00
Pekka Enberg
d8f07fe3da core: Panic on fsync() error by default
Retrying fsync() on error was historically not safe ("fsyncgate") and
Postgres still defaults to panicing on fsync(). Therefore, add a
"data_sync_retry" pragma (disabled by default) and use it to determine
whether to panic on fsync() error or not.
2025-09-13 10:21:12 +03:00
Pekka Enberg
a7e34f1551 Merge 'Handle partial writes in unix IO for pwrite and pwritev' from Preston Thorpe
currently, `io_uring` is setup to handle partial writes for `pwritev`
(will add `pwrite` in subsequent PR), but unix and other IO back-ends
were not correctly setup for this.

Closes #3073
2025-09-13 09:08:43 +03:00
Pekka Enberg
f547a302cc Merge 'remove Stmt clone' from Lâm Hoàng Phúc
Closes #3076
2025-09-13 09:06:15 +03:00
Avinash Sajjanshetty
5256f29a9c Add checksums behind a feature flag 2025-09-13 11:00:39 +05:30
Avinash Sajjanshetty
06a824ec68 Add checksum tests 2025-09-13 11:00:39 +05:30
Avinash Sajjanshetty
1536f65f07 move test helper run_query to common module 2025-09-13 11:00:39 +05:30