Pekka Enberg
1f79fbc22c
Merge 'Partial sync basic' from Nikita Sivukhin
...
This PR implements basic support for partial sync. Right now the scope
is limited to only `:memory:` IO and later will be properly expanded to
the file based IO later.
The main addition is `PartialDatabaseStorage` which make request to the
remote server for missing local pages on demand.
The main change is that now tursodatabase JS bindings accept optional
"external" IO event loop which in case of sync will drive `ProtocolIo`
internal work associated with remote page fetching tasks.
Closes #3931
2025-11-13 16:38:04 +02:00
Nikita Sivukhin
740ff2b4a6
fix clippy
2025-11-12 16:46:15 +04:00
Nikita Sivukhin
15dafd46c1
replace turso_assert -> assert
2025-11-12 16:40:38 +04:00
Nikita Sivukhin
3d14092679
fix
2025-11-12 16:38:04 +04:00
Nikita Sivukhin
72089d2682
adjust compilation
2025-11-12 16:30:50 +04:00
Nikita Sivukhin
aa65cfd55d
update Cargo.toml
2025-11-12 16:14:14 +04:00
Nikita Sivukhin
a25e3e76eb
wip
2025-11-12 13:21:34 +04:00
Nikita Sivukhin
6f7edcaddd
agent review fixes
2025-11-12 12:32:45 +04:00
Nikita Sivukhin
b73ff13b88
add simple implementation of Sparse IO
2025-11-12 12:04:12 +04:00
Nikita Sivukhin
d519945098
make ArenaBuffer unsafe Send + Sync
2025-11-12 10:54:40 +04:00
Nikita Sivukhin
33375697d1
add partial database storage implementation
2025-11-12 10:53:25 +04:00
Nikita Sivukhin
a855a657aa
report network stats
2025-11-12 10:53:25 +04:00
Nikita Sivukhin
98db727a99
integrate extra io stepping logic to the JS bindings
2025-11-12 10:53:25 +04:00
Nikita Sivukhin
d42b5c7bcc
wip
2025-11-12 10:53:25 +04:00
Nikita Sivukhin
34f1072071
add hooks to plug partial sync in the sync engine
2025-11-12 10:53:25 +04:00
pedrocarlo
1db13889e3
Change Value::Text to use a Cow<'static, str> instead of Vec<u8>
2025-11-11 16:11:46 -03:00
Nikita Sivukhin
da61fa32b4
use dyn DatabaseStorage instead of DatabaseFile
2025-11-06 17:42:03 +04:00
Nikita Sivukhin
d013876998
cargo fmt
2025-10-29 16:46:51 +04:00
Nikita Sivukhin
82d54999b1
fix pull operation in sync engine
...
- before we fetched pull generation and last_change_id from the remote during pull - which is incorrect because fetched information can be inconsistent with WAL updates we received from the server (latest server state can be in "future" compared to the WAL updates we got since we can make push in parallel with updates pull operation)
- now we read information about "server state" (pull generation, last_change_id) directly from the local DB right after we applied changes from the remote which get us consistent view on the state considering WAL updates we got
- also fetching remote in the pull is bad - since pull block writes and network call with unpredictable latency poorly affect writes to the database
2025-10-29 16:24:05 +04:00
pedrocarlo
5a7390735d
rename Completion functions
2025-10-06 11:07:06 -03:00
pedrocarlo
e93add6c80
remove dyn DatabaseStorage and replace it with DatabaseFile
2025-10-03 14:14:15 -03:00
Nikita Sivukhin
5ec2d96bc1
support defered sync in the engine
2025-10-02 16:17:36 +04:00
Nikita Sivukhin
7016d9456e
fix test in the database tape
2025-10-02 16:17:36 +04:00
Nikita Sivukhin
91aeadd940
apply create DDL operation with IF NOT EXISTS clause in order to make them idempotent
2025-10-02 15:56:34 +04:00
Pavan-Nambi
49d5141f2d
Merge remote-tracking branch 'origin/main' into cdc_fail_autoincrement
2025-09-24 18:06:02 +05:30
Nikita Sivukhin
57c3d783e1
fix clippy
2025-09-23 18:24:52 +04:00
Nikita Sivukhin
e89dac98f3
use hrana batch stmt with Not(IsAutocommit) condition for push operation
2025-09-23 16:33:40 +04:00
Pavan Nambi
f1ac855441
Merge branch 'main' into cdc_fail_autoincrement
2025-09-22 21:11:26 +05:30
Nikita Sivukhin
1185298670
fix replay generator
2025-09-19 13:19:30 +04:00
Nikita Sivukhin
160119b12e
propagate long poll timeout
2025-09-19 13:19:30 +04:00
Nikita Sivukhin
b0f60a29ba
guard meta with mutex and make all methods to accept shared reference and offload locking of sync engine methods to the external wrapper
2025-09-19 13:19:30 +04:00
Pavan-Nambi
020921f803
Merge remote-tracking branch 'upstream/main' into cdc_fail_autoincrement
2025-09-18 19:27:19 +05:30
rajajisai
e605aff31b
Merge branch 'main' into enc-page-1
2025-09-16 10:06:00 -04:00
rajajisai
89caa868f9
Encryption support for database header page
2025-09-16 10:04:30 -04:00
pedrocarlo
3c91ae206b
move as many dependencies as possible to workspace to avoid multiple versions of the same dependency
2025-09-15 17:19:36 -03:00
Nikita Sivukhin
9b5656d4dc
fix stats method
2025-09-15 11:05:49 +04:00
Nikita Sivukhin
527d0cb1f3
expose revision in the stats method
2025-09-15 10:56:13 +04:00
Pavan-Nambi
f5c52065ed
update sync-engine tests
2025-09-13 07:10:18 +05:30
Nikita Sivukhin
1a4d334e47
fix clippy and remove old action yaml
2025-09-11 00:03:01 +04:00
Nikita Sivukhin
d55026f84f
opfs for sync in one commit!
2025-09-10 22:35:57 +04:00
Pekka Enberg
2131a04b7d
core: Rename IO::run_once() to IO::step()
...
The `run_once()` name is just a historical accident. Furthermore, it now
started to appear elsewhere as well, so let's just call it IO::step() as we
should have from the beginning.
2025-09-10 14:36:02 +03:00
Pekka Enberg
12cf4d2e72
core: Make strict schema support experimental
...
It's not tested properly so let's mark it as experimental for now.
Fixes #2775
2025-09-02 16:40:02 +03:00
PThorpe92
fb7e6fb280
Guard against all overflow when calculating IO offsets
2025-08-28 09:50:26 -04:00
PThorpe92
0a56d23402
Use u64 for file offsets in IO and calculate such offsets in u64
2025-08-28 09:44:00 -04:00
Nikita Sivukhin
b67f14c785
fix clippy
2025-08-27 15:57:38 +04:00
Nikita Sivukhin
6e124d927e
fix clippy
2025-08-27 15:51:29 +04:00
Nikita Sivukhin
009aa479bf
improve sync engine
2025-08-27 15:30:00 +04:00
Nikita Sivukhin
94ed92ca4e
fix compilation and clippy
2025-08-21 18:28:16 +04:00
Nikita Sivukhin
c34d884b6e
fix tests
2025-08-21 15:24:21 +04:00
Nikita Sivukhin
abe3404953
fix build
2025-08-21 15:15:23 +04:00
