Commit Graph

7580 Commits

Author SHA1 Message Date
Nikita Sivukhin
34a7b2ffd4 ignore changes in the turso_sync_last_change_id 2025-08-14 12:39:44 +04:00
Nikita Sivukhin
8c9d648852 add test which check that we don't push without the need 2025-08-14 12:38:15 +04:00
Nikita Sivukhin
f603a0dfc8 change log level to INFO in order to simplify debugging (DEBUG logs in the db are pretty spammy) 2025-08-14 12:37:49 +04:00
Nikita Sivukhin
887b25dd00 do not push wal unnecessary when nothing was changed locally 2025-08-13 20:22:10 +04:00
Pekka Enberg
aaf7b39d96 Update manual.md 2025-08-13 18:52:14 +03:00
Pekka Enberg
29aea48405 Merge 'Document the I/O model' from Pedro Muniz
I modified the `docs/manual.md` file to store the current I/O model that
Turso follows. I want some feedback on the text and on the design as
well!

Closes #2308
2025-08-13 18:50:58 +03:00
Pekka Enberg
c62a9558e2 Merge 'Sync engine fixes' from Nikita Sivukhin
This PR fixes several small bugs around the sync-engine:
1. WAL pull handled end of the "frame" iterator incorrectly - this is
fixed in the https://github.com/tursodatabase/turso/commit/eff8d8540d1e8
3214459822ac6eeb0d3409ecc24 and test with concurrent DBs were added
2. Using `:memory:` in the sync engine lead to weird behavior because
engine will create different `MemoryIO` instances but turso-core under
the hood will use global registry of databases. I **changed** criteria
for determining in-memory databases by checking the prefix of the path
to be equal to `:memory:` (https://github.com/tursodatabase/turso/commit
/80476b3069f2cd460f368e10b4b2ef51f7608077)
3. Switched from `Buffer` to `Vec<u8>` for now as browser seems to not
support `Buffer` natively: https://github.com/tursodatabase/turso/commit
/2ca8a15dcc8ec79eddfaf7068fab4e1aa3241506
4. Added tracing to the sync engine: https://github.com/tursodatabase/tu
rso/commit/33ef1aa0da7dfe9416025e26e98f9cf9d48c9119

Closes #2582
2025-08-13 18:49:36 +03:00
Pekka Enberg
fbab40e5a6 Merge 'disable checkpoint: adjust semantic' from Nikita Sivukhin
This PR change semantic of `wal_disable_checkpoint` flag and make it
disable only "automatic" checkpoint behaviour which at the moment
includes:
1. Checkpoint on shutdown
2. Checkpoint when WAL reach certain size
sync-engine needs checkpoint to be in full control but it also will
issue `TRUNCATE` checkpoint at some certain moments of time - so
complete disablement of checkpoints will not suffice.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2580
2025-08-13 18:49:11 +03:00
Pekka Enberg
753e6689da Merge 'SDK: enable indices everywhere' from Nikita Sivukhin
Indices enabled by default in the shell - so let's enable them also in
the SDK

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #2581
2025-08-13 18:44:41 +03:00
pedrocarlo
e416d91df2 Proposed I/O model 2025-08-13 12:16:14 -03:00
Nikita Sivukhin
56b86cd5f5 add comment about :memory: in sync-engine 2025-08-13 17:16:46 +04:00
Nikita Sivukhin
eff8d8540d fix bug and add test with concurrent dbs in sync 2025-08-13 17:08:07 +04:00
Nikita Sivukhin
80476b3069 bypass database registry for all dbs which path starts with :memory: prefix
- sync engine create pair of databases and they must be isolated but live in the same MemoryIO
- the problem can happen if there will be 2 sync engines with MemoryIO storage layer - as they all will create :memory:-draft and :memory:-synced DBs
2025-08-13 17:00:01 +04:00
Nikita Sivukhin
615207fb9c use proper event loop in core connection in order to handle all cases properly
- otherwise, in case of schema change, connection will constantly get Database schema is changed error as reprepare logic is implemented in the statement event loop
2025-08-13 17:00:01 +04:00
Nikita Sivukhin
2ca8a15dcc switch from Buffer to Vec<u8> for now as buffers are not supported by default in browser 2025-08-13 17:00:01 +04:00
Nikita Sivukhin
33ef1aa0da add tracing option to the SyncEngine 2025-08-13 17:00:01 +04:00
Nikita Sivukhin
0d9d0762a9 fix clippy 2025-08-13 16:57:44 +04:00
Nikita Sivukhin
325e990d6e fix clippy 2025-08-13 16:55:23 +04:00
Nikita Sivukhin
6b97f2584f enable indices in Dart SDK 2025-08-13 16:47:14 +04:00
Nikita Sivukhin
d1eab18670 enable indices in java sdk 2025-08-13 16:46:25 +04:00
Nikita Sivukhin
0f1d954319 enable indices in Rust SDK 2025-08-13 16:28:45 +04:00
Nikita Sivukhin
70581aca5b enable indices in go sdk 2025-08-13 16:24:04 +04:00
Nikita Sivukhin
857f9147f6 enable indices in the python SDK 2025-08-13 16:10:27 +04:00
Nikita Sivukhin
bfa33a27e2 enable indices in database JS sdk 2025-08-13 15:55:40 +04:00
Nikita Sivukhin
734750daaa fix after rename 2025-08-13 15:26:25 +04:00
Nikita Sivukhin
5838efe7dd rename flag to wal_auto_checkpoint_disabled 2025-08-13 15:26:25 +04:00
Nikita Sivukhin
f9f56f454c change semantic of wal_disable_checkpoint flag - it disable only automatic checkpoint (on shutdown or when WAL is grows too big) 2025-08-13 15:23:18 +04:00
Nikita Sivukhin
0a12b2d74a disable auto-checkpoint for opened connections in the sync engine 2025-08-13 15:23:18 +04:00
Nikita Sivukhin
6e0ad0a483 open draft database only once
- otherwise there will be problems with memory IO backend
2025-08-13 15:23:18 +04:00
Nikita Sivukhin
a78992e309 add assertion with more context 2025-08-13 15:23:18 +04:00
Nikita Sivukhin
c3c7463193 add debug logs to the memory io 2025-08-13 15:23:18 +04:00
Pekka Enberg
57cedcac72 Fix JavaScript getting started 2025-08-13 13:26:40 +03:00
Pekka Enberg
901191d1ee Update README.md 2025-08-13 13:24:00 +03:00
Pekka Enberg
31b826cca7 Merge 'Manual updates' from Pekka Enberg
Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2578
2025-08-13 13:17:50 +03:00
Pekka Enberg
09d64b2290 docs/manual: Drop redundant section place-holder
There's already a section on SQL query language.
2025-08-13 12:55:56 +03:00
Pekka Enberg
c942239a5a docs/manual: Document more of SQLite C API 2025-08-13 12:54:37 +03:00
Pekka Enberg
770b43caa6 docs/manual: Drop indexing from list of limitations
Indexes are enabled by default now.
2025-08-13 12:47:21 +03:00
Pekka Enberg
379896d087 docs/manual: Document some shell commands 2025-08-13 12:46:17 +03:00
Pekka Enberg
bbc7994f59 docs/manual: Document shell command line options 2025-08-13 12:42:48 +03:00
Pekka Enberg
ece8f106d9 docs/manual: Fix JavaScript section
The API changed so fix that up. Also improve flow of text.
2025-08-13 12:26:31 +03:00
Pekka Enberg
9a0d5cebe9 docs/manual: Fix limitations section heading style 2025-08-13 12:26:11 +03:00
Pekka Enberg
db0d80310f Update manual.md 2025-08-13 11:57:47 +03:00
Pekka Enberg
c856472dbe docs: Add a section on MVCC internals 2025-08-13 11:52:02 +03:00
Jussi Saurio
1a625e7bf6 Merge 'Wait for I/O completions' from Pedro Muniz
Depends on #2566.
This PR bubbles all completions up to the `step()` function. Then in the
step function we store the completions and before proceeding with the
next instruction, we first check if IO was completed. Because of this
feature, we do not have to build most our code around waiting for IO to
complete or for the page to be loaded, as we should not proceed to next
state if we did not await all completions.

Closes #2570
2025-08-13 11:23:08 +03:00
pedrocarlo
187ffb5698 adjust test 2025-08-13 10:24:55 +03:00
pedrocarlo
8d48be0e61 cleanup 2025-08-13 10:24:55 +03:00
pedrocarlo
fbe7e685ce adjust mvcc code to return completions in state machines 2025-08-13 10:24:55 +03:00
pedrocarlo
c1975cdfa1 ptrmap functions should bubble completions 2025-08-13 10:24:55 +03:00
pedrocarlo
b6e200dbed adjust cacheflush calls outside of pager 2025-08-13 10:24:55 +03:00
pedrocarlo
78cb61c1fe before stepping to next insntruction check for io 2025-08-13 10:24:55 +03:00