Commit Graph

8857 Commits

Author SHA1 Message Date
pedrocarlo
bd5dcd8d3c add timeout flag to throughput benchmark 2025-09-15 02:20:32 -03:00
pedrocarlo
3d265489dc modify semantics of busy_timeout to be more on par with sqlite 2025-09-15 02:20:32 -03:00
pedrocarlo
0586b75fbe expose function to set busy timeout duration 2025-09-15 02:20:32 -03:00
pedrocarlo
16e79ed508 slight adjustment in perf throughtput printing 2025-09-15 02:16:18 -03:00
pedrocarlo
a56680f79e implement Busy Handler in Turso statements 2025-09-15 02:16:18 -03:00
Pekka Enberg
24c50597ad Merge 'Random fixes for MVCC' from Jussi Saurio
based on #3110
closes #3111
closes #3113
closes #3114
all discovered using `test_multiple_connections_fuzz_mvcc`, so no
separate tests. i can add regression unit/integration tests tomorrow
    mvcc: properly remove mutations of rolled back tx
    mvstore was not removing deletions made by a tx that rolled back.
    deletions are removed by clearing the `end` mark from the row
    version.
---
    mvcc: properly clear tx states when mvcc tx rolls back
---
    mvcc: don't double-rollback on write-write-conflict
    handle_program_error() already rolls back if this error happens.
    double rollback causes a crash.

Closes #3115
2025-09-15 07:39:35 +03:00
Pekka Enberg
27d4572c2e Merge 'mvcc: Complete commit state machine early if write set is empty' from Jussi Saurio
Closes #3104
2025-09-15 07:38:28 +03:00
Jussi Saurio
f4c15a37d3 add manual hack to mvcc test
we rollback the mvcc transaction in the VDBE, so manually roll it
back in the test
2025-09-14 23:46:38 +03:00
Jussi Saurio
db3428a7a9 remove unused pager parameter 2025-09-14 23:44:24 +03:00
Jussi Saurio
d598775e33 mvcc: properly remove mutations of rolled back tx
mvstore was not removing deletions made by a tx that rolled back.
deletions are removed by clearing the `end` mark from the row
version.
2025-09-14 23:29:14 +03:00
Jussi Saurio
dccf8b9472 mvcc: properly clear tx states when mvcc tx rolls back 2025-09-14 23:29:07 +03:00
Jussi Saurio
487b8710d9 mvcc: don't double-rollback on write-write-conflict
handle_program_error() already rolls back if this error happens.
double rollback causes a crash.
2025-09-14 23:28:21 +03:00
Jussi Saurio
8510721c5e Merge 'add perf/throughput/turso to workspace' from Pedro Muniz
Closes #3102
2025-09-14 22:30:49 +03:00
Jussi Saurio
2ca1640a2a not always write 2025-09-14 22:24:07 +03:00
pedrocarlo
01a99f84a6 add perf/throughput/turso to workspace 2025-09-14 16:19:34 -03:00
Jussi Saurio
396091044e store tx_mode in conn.mv_tx
otherwise op_transaction works completely wrong because each separate
insert statement overrides the tx_mode to Write
2025-09-14 21:59:08 +03:00
Jussi Saurio
7fe25a1d0e mvcc: remove conn.mv_transactions
afaict this isn't needed for anything since there is already
conn.mv_tx_id
2025-09-14 21:26:58 +03:00
Jussi Saurio
5feb9ea2f0 mvcc: fix non-concurrent transaction semantics
on the main branch, mvcc allows concurrent inserts from multiple
txns even without BEGIN CONCURRENT, and then always hangs whenever
one of the txns tries to commit.

this commit fixes that issue.
2025-09-14 21:23:06 +03:00
Jussi Saurio
2ea1798d6e mvcc: end commit state machine early when write set is empty 2025-09-14 20:02:35 +03:00
Pekka Enberg
bc4aa63203 Merge 'make whopper run with checksums' from Avinash Sajjanshetty
pass `--enable-checksums` to either `run` or `explore` commands

Closes #3100
2025-09-14 18:26:12 +03:00
Pekka Enberg
76a3c20a48 Merge 'perf/throughput/turso: Don't use spawn_blocking()' from Pekka Enberg
Let's just use normal spawn()

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #3096
2025-09-14 17:55:44 +03:00
Pekka Enberg
3273d78969 Merge 'Whopper + MVCC' from Pekka Enberg
Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #3097
2025-09-14 17:55:25 +03:00
Avinash Sajjanshetty
d35789690e make whopper run with checksums
pass `--enable-checksums` to either `run` or `explore` commands
2025-09-14 20:08:34 +05:30
Pekka Enberg
1bdfabcac7 whopper: Generate different transaction modes with MVCC 2025-09-14 13:55:08 +03:00
Pekka Enberg
5783e3edf8 whopper: Add "--enable-mvcc" command line option 2025-09-14 13:15:52 +03:00
Pekka Enberg
db190c9e3d whopper: Add "--disable-indexes" option
MVCC does not support indexes so let's add an option to disable that.
2025-09-14 13:13:49 +03:00
Pekka Enberg
95660535da core/storage: Demote info logging to debug 2025-09-14 13:10:46 +03:00
Pekka Enberg
a5f1bdbc9d Merge 'perf/throughput/turso: Don't interleave concurrent transactions in on…' from Pekka Enberg
…e connection
With BEGIN CONCURRENT, we cannot use the same connection all the time
because there's no way for the transaction manager to know which
transaction belongs to what "session" -- they're all individual
statements executed in the context of the one connection.
Fixes #3093

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3094
2025-09-14 11:33:20 +03:00
Pekka Enberg
ed8b02d83e perf/throughput/turso: Don't use spawn_blocking()
Let's just use normal spawn()
2025-09-14 11:32:17 +03:00
Pekka Enberg
222b9e7ca2 Merge 'perf/throughput/turso: Don't execute futures serially' from Pekka Enberg
Looping through the futures means we're essentially executing them
serially.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3092
2025-09-14 11:01:06 +03:00
Pekka Enberg
6e9ba4c561 perf/throughput/turso: Don't interleave concurrent transactions in one connection
With BEGIN CONCURRENT, we cannot use the same connection all the time
because there's no way for the transaction manager to know which
transaction belongs to what "session" -- they're all individual
statements executed in the context of the one connection.

Fixes #3093
2025-09-14 10:52:56 +03:00
Pekka Enberg
98af9386e2 perf/throughput/turso: Don't execute futures serially
Looping through the futures means we're essentially executing them
serially.
2025-09-14 10:07:21 +03:00
Preston Thorpe
4e4f0ea61f Merge 'Dont grab page cache write lock in a loop' from Preston Thorpe
durp

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3088
2025-09-13 12:48:19 -04:00
PThorpe92
f6dd0bc4d6 Dont grab page cache write lock in a loop 2025-09-13 12:21:13 -04:00
Pekka Enberg
6a2f0d6061 Merge 'Add per page checksums' from Avinash Sajjanshetty
This patch adds checksums to Turso DB. You may check the design here in
the [RFC](https://github.com/tursodatabase/turso/issues/2178).
1. We use reserved bytes (8 bytes) to store the checksums. On every IO
read, we verify that the checksum matches.
2. We use twox hash for checksums.
3. Checksum works only on 4K pages now. It's a small change to enable
for all other sizes, I will send another PR.
4. Right now, it's not possible to switch to different algorithm or turn
off altogether. That will be added in the future PRs.
5. Checksums can be enabled only for new dbs. For existing DBs, we will
disable it.
6. To add checksums for existing DBs, we need vacuum since it would
require rewrite of whole db.

Closes #2840
2025-09-13 18:46:53 +03:00
Pekka Enberg
7f5038f7c9 Merge 'perf/throughput/turso: Async transactions with concurrent mode' from Pekka Enberg
With `BEGIN CONCURRENT`, we should also take advantage of async
transaction processing to maximize concurrency.

Closes #3082
2025-09-13 15:07:29 +03:00
Pekka Enberg
7d3ce68695 Merge 'core/throughput: Add per transaction think time support' from Pekka Enberg
Closes #3080
2025-09-13 14:07:30 +03:00
Pekka Enberg
898f32f7f7 Fix Antithesis Dockerfile to include whopper 2025-09-13 13:33:11 +03:00
Pekka Enberg
3733d3856a Merge 'core: Panic on fsync() error by default' from Pekka Enberg
Retrying fsync() on error was historically not safe ("fsyncgate") and
Postgres still defaults to panicing on fsync(). Therefore, add a
"data_sync_retry" pragma (disabled by default) and use it to determine
whether to panic on fsync() error or not.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3077
2025-09-13 13:32:07 +03:00
Pekka Enberg
0fad30a30d perf/throughput/turso: Async transactions with concurrent mode
With `BEGIN CONCURRENT`, we should also take advantage of async
transaction processing to maximize concurrency.
2025-09-13 13:25:16 +03:00
Pekka Enberg
8dc2e738a4 core/throughput: Add per transaction think time support 2025-09-13 13:02:43 +03:00
Pekka Enberg
d8f07fe3da core: Panic on fsync() error by default
Retrying fsync() on error was historically not safe ("fsyncgate") and
Postgres still defaults to panicing on fsync(). Therefore, add a
"data_sync_retry" pragma (disabled by default) and use it to determine
whether to panic on fsync() error or not.
2025-09-13 10:21:12 +03:00
Pekka Enberg
a7e34f1551 Merge 'Handle partial writes in unix IO for pwrite and pwritev' from Preston Thorpe
currently, `io_uring` is setup to handle partial writes for `pwritev`
(will add `pwrite` in subsequent PR), but unix and other IO back-ends
were not correctly setup for this.

Closes #3073
2025-09-13 09:08:43 +03:00
Pekka Enberg
f547a302cc Merge 'remove Stmt clone' from Lâm Hoàng Phúc
Closes #3076
2025-09-13 09:06:15 +03:00
Avinash Sajjanshetty
5256f29a9c Add checksums behind a feature flag 2025-09-13 11:00:39 +05:30
Avinash Sajjanshetty
06a824ec68 Add checksum tests 2025-09-13 11:00:39 +05:30
Avinash Sajjanshetty
1536f65f07 move test helper run_query to common module 2025-09-13 11:00:39 +05:30
Avinash Sajjanshetty
11030056c7 rename method to verify_checksum 2025-09-13 11:00:39 +05:30
Avinash Sajjanshetty
e010c46552 use checksums when reading/writing from db file 2025-09-13 11:00:39 +05:30
Avinash Sajjanshetty
4b59cf19e5 use checksums when reading/writing from wal 2025-09-13 11:00:39 +05:30