Commit Graph

8162 Commits

Author SHA1 Message Date
PThorpe92
0a56d23402 Use u64 for file offsets in IO and calculate such offsets in u64 2025-08-28 09:44:00 -04:00
Pekka Enberg
2ea4354afe Merge 'Improve integrity check' from Nikita Sivukhin
- check free list trunk and pages
- use shared hash map to check for duplicate references for pages
- properly check overflow pages

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2816
2025-08-28 16:06:15 +03:00
Pekka Enberg
878147b931 Merge 'translate/insert: Improve string format performance' from Pere Diaz Bou
Rust's `fmt!` is slow af for the simplest of cases, let's just create
strings with a known size and skip all the fmt stuff.

Closes #2832
2025-08-28 14:36:09 +03:00
Pekka Enberg
a5322d2ab7 Merge 'core/schema: get_dependent_materialized_views_unnormalized' from Pere Diaz Bou
If we get a table name for in memory structure, it's safe to assume it's
already normalized.

Closes #2830
2025-08-28 14:34:06 +03:00
Pekka Enberg
eba90e1c5e Merge 'core/util: emit literal, cow instead of replace' from Pere Diaz Bou
Closes #2828
2025-08-28 14:33:04 +03:00
Pekka Enberg
44e619be1a Merge 'core/translate: sanize_string fast path improvement' from Pere Diaz Bou
Closes #2827
2025-08-28 14:32:43 +03:00
Pere Diaz Bou
964422375e translate/insert: string fmt perf improvmenets 2025-08-28 13:22:54 +02:00
Pere Diaz Bou
48e5ad7a55 core/schema: get_dependent_materialized_views_unnormalized
If we get a table name for in memory structure, it's safe to assume it's
already normalized.
2025-08-28 13:11:40 +02:00
Pere Diaz Bou
84c5c4e581 core/util: emit literal, cow instead of replace 2025-08-28 12:59:55 +02:00
Pere Diaz Bou
082f18c073 core/translate: sanize_string fast path improvement 2025-08-28 12:57:28 +02:00
Pekka Enberg
b4eba8b456 Merge 'Fix planner alias and table name handling' from
closes: #2817
```
turso> select * from simple_view as "v" where "v"."x" = 1;
┌───┐
│ x │
├───┤
│ 1 │
└───┘
turso> select "s"."x" from simple as "s" where "s"."x" = 1;
┌───┐
│ x │
├───┤
│ 1 │
└───
```

Reviewed-by: Preston Thorpe <preston@turso.tech>
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2818
2025-08-28 07:39:47 +03:00
Pekka Enberg
591914a63b Merge 'Introduce and propagate IOContext as required' from Avinash Sajjanshetty
I added `IOContext` to `DatbaseStorage` IO trait and this struct will
carry the necessary ctx required for encryption (or checksums.). This
lets us set the encryption at outside and let the IO layer handle it
properly

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2812
2025-08-28 07:38:52 +03:00
Pekka Enberg
20260c9dc0 Merge 'core/io: Switch Unix I/O to use libc::pwrite()' from Pekka Enberg
We use libc elsewhere for fault injection reasons, so let's do this
call-site too.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2811
2025-08-28 07:38:42 +03:00
Pekka Enberg
376dbc9261 Merge 'Update README.md for Go documentation' from Preston Thorpe
Closes #2819
2025-08-28 07:38:20 +03:00
PThorpe92
1526448089 Update README.md for Go documentation 2025-08-27 22:23:19 -04:00
themixednuts
79a9f4743e fix: planner alias and table name 2025-08-27 18:13:03 -05:00
Nikita Sivukhin
1c0efcfbff fix clippy 2025-08-27 23:22:21 +04:00
Nikita Sivukhin
09d4590ece fix compilation 2025-08-27 23:19:26 +04:00
Nikita Sivukhin
ae705445bf improve integrity check
- check free list trunk and pages
- use shared hash map to check for duplicate references for pages
- properly check overflow pages
2025-08-27 23:14:21 +04:00
Avinash Sajjanshetty
2c0842ff52 Set and propagate IOContext as required 2025-08-27 22:05:01 +05:30
Avinash Sajjanshetty
9e663c7f46 Add IOContext to carry encryption/checksum ctx 2025-08-27 21:33:05 +05:30
Pekka Enberg
e6ae7f990d Merge 'improve sync engine' from Nikita Sivukhin
This PR makes plenty of changes to the sync engine among which the main
ones are:
1. Now, we have only single database file which we use to directly
execute `pull` and `push` requests (so we don't have `draft` / `synced`
databases)
2. Last-write-win strategy were fixed a little bit - because before this
PR insert-insert conflict wasn't resolved automatically
3. Now sync engine can apply arbitrary `transform` function to the
logical changes
4. Sync-engine-aware checkpoint was implemented. Now, database created
by sync-engine has explicit `checkpoint()` method which under the hood
will use additional file to save frames needed for revert operation
during pull
5. Pull operation were separated into 2 phases internally: wait for
changes & apply changes
    * The problem is that pull operation itself (e.g. apply) right now
require exclusive lock to the sync engine and if user wants to pull &
push independently this will be problematic (as pull will lock the db
and push will never succeed)
6. Added support for V1 pull protocol
7. Exposed simple `stats()` method which return amount of pending cdc
operations and current wal size

Closes #2810
2025-08-27 18:08:21 +03:00
Pekka Enberg
bf7f80a937 core/io: Switch Unix I/O to use libc::pwrite()
We use libc elsewhere for fault injection reasons, so let's do this
call-site too.
2025-08-27 17:56:23 +03:00
Nikita Sivukhin
b67f14c785 fix clippy 2025-08-27 15:57:38 +04:00
Nikita Sivukhin
6e124d927e fix clippy 2025-08-27 15:51:29 +04:00
Nikita Sivukhin
009aa479bf improve sync engine 2025-08-27 15:30:00 +04:00
Pekka Enberg
30c5473151 Merge 'Add some docs on encryption' from Avinash Sajjanshetty
Reviewed-by: Preston Thorpe <preston@turso.tech>
Reviewed-by: bit-aloo (@Shourya742)

Closes #2805
2025-08-27 13:27:00 +03:00
Pekka Enberg
73ef26125b Merge 'Remove Go bindings' from Preston Thorpe
Go bindings have moved to their own repo
https://github.com/tursodatabase/turso-go
Otherwise it is a pain in the ass to have a Go module as a subdirectory
of another repository. and now, they can get the TLC that they needed,
as I found out that it never worked for windows, and the embedding
wasn't working properly in some cases. We want them to be plug and play.

Closes #2807
2025-08-27 13:26:45 +03:00
Pekka Enberg
eced1fe7db Merge 'core/storage: Micro-optimize Pager::commit_dirty_pages()' from Pekka Enberg
There's no need to call io.now() unless debug tracing is on. Let's
micro-optimize commit_dirty_pages() to avoid the unnecessary call.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #2808
2025-08-27 13:26:22 +03:00
Pekka Enberg
2921033b28 core/storage: Micro-optimize Pager::commit_dirty_pages()
There's no need to call io.now() unless debug tracing is on. Let's
micro-optimize commit_dirty_pages() to avoid the unnecessary call.
2025-08-27 11:12:43 +03:00
PThorpe92
37f71b84c8 Remove go github workflow 2025-08-26 19:41:53 -04:00
PThorpe92
bcdcd47358 Remove Go bindings (moved to their own repo tursodatabase/turso-go) 2025-08-26 19:13:17 -04:00
Preston Thorpe
d122105f8c Merge 'Rename Go driver to turso to not conflict with sqlite3' from Preston Thorpe
Also rename Limbo -> Turso while we are here..

Closes #2806
2025-08-26 14:32:09 -04:00
PThorpe92
2614a42294 Update package name in go CI 2025-08-26 14:18:57 -04:00
PThorpe92
4cf111e3c2 Rename Go driver to turso to not conflict with sqlite3, rename limbo->turso 2025-08-26 14:13:42 -04:00
Avinash Sajjanshetty
4d7b4bb711 Add some docs on encryption 2025-08-26 23:11:48 +05:30
Preston Thorpe
77476de547 Merge 'Refactor: Cell instead of RefCell to store CipherMode in connection' from Avinash Sajjanshetty
follow up PR for the review here:
https://github.com/tursodatabase/turso/pull/2779#discussion_r2299460752

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2804
2025-08-26 11:39:05 -04:00
Avinash Sajjanshetty
caa00e31f8 Use Cell instead of RefCell because its nice 2025-08-26 20:00:13 +05:30
Jussi Saurio
66d00915d7 Merge 'Improve documentation of page pinning' from Jussi Saurio
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2797
2025-08-26 17:18:25 +03:00
Jussi Saurio
bf98bf4576 Merge 'Fix missing functions after revert' from Pedro Muniz
When we reverted the #2789 , we had already merged #2793 . That Pr used
some helper methods that were created in #2789. So I just added them
back here + fixed the simulator dockerfile.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2803
2025-08-26 17:18:06 +03:00
pedrocarlo
aa025c9798 fix missing functions after revert 2025-08-26 10:13:45 -03:00
Pekka Enberg
26ba09c45f Revert "Merge 'Remove double indirection in the Parser' from Pedro Muniz"
This reverts commit 71c1b357e4, reversing
changes made to 6bc568ff69 because it
actually makes things slower.
2025-08-26 14:58:21 +03:00
Pekka Enberg
b7bf4b55ed Merge 'Fail CI run if Turso output differs from SQLite in TPC-H queries' from Jussi Saurio
We had missed #2794 because we don't get any visible feedback on PRs if
we return wrong results from TPC-H, so let's make that happen here.

Closes #2798
2025-08-26 14:36:22 +03:00
Jussi Saurio
e65742e5ff Fail CI if tursodb output differs from sqlite in tpc-h queries 2025-08-26 11:30:37 +03:00
Jussi Saurio
bf58d179db Improve documentation of page pinning 2025-08-26 10:13:25 +03:00
Pekka Enberg
5dd1bca4d3 Merge 'Decouple SQL generation from Simulator crate' from Pedro Muniz
Decouple Sql generation code from simulator code, so that it can
potentially be reused for fuzzing on other crates and to create a
`GenerationContext` trait so that it becomes easier to create
`Simulation Profiles`. Ideally in further PRs, I want to expand the
`GenerationContext` trait so we can guide the generation with context
from the simulation profile.
Depends on #2789 .

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2793
2025-08-26 09:41:58 +03:00
Pekka Enberg
3176df64a2 Merge 'Fix: return NULL for rowid() when cursor's null flag is on' from Jussi Saurio
Fixes TPC-H query 13 from returning an incorrect result. In this
specific case, we were returning non-null `IdxRowid` values for the
right-hand side table even when there was no match with the left-hand
side table, meaning the join produced matches even in cases where there
shouldn't have been any.
Closes #2794

Closes #2795
2025-08-26 09:33:49 +03:00
Jussi Saurio
3905f0af46 Add regression test for issue 2794 2025-08-26 09:21:58 +03:00
Jussi Saurio
e52f807c7d Fix: return NULL for rowid() when cursor's null flag is on
Fixes TPC-H query 13 from returning an incorrect result. In this specific
case, we were returning non-null `IdxRowid` values for the right-hand side
table even when there was no match with the left-hand side table, meaning
the join produced matches even in cases where there shouldn't have been any.

Closes #2794
2025-08-26 09:08:48 +03:00
Pekka Enberg
ec73b809a9 antithesis-tests: Enable multi-threading 2025-08-26 08:37:35 +03:00