on the main branch, mvcc allows concurrent inserts from multiple
txns even without BEGIN CONCURRENT, and then always hangs whenever
one of the txns tries to commit.
this commit fixes that issue.
Adds the AggValue instruction, which computes the current aggregate
result and writes it to a dedicated destination register.
Unlike AggFinal, it does not overwrite or clear the accumulator
register. This makes it possible to retrieve aggregate results multiple
times—needed when processing window functions—while preserving the
accumulator state.
Previously, only the External and Avg aggregates mutated state during
AggFinal. This is unnecessary because AggFinal runs only once per group,
so caching the result provides no performance benefit.
By avoiding state mutation, we can also reuse op_agg_final for the
AggValue instruction that will be added soon.
The transaction upgrade logic in Transaction opcode is total nonsense
for concurrent transactions so just drop it.
Fixes#3061
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#3070
Fixes#1817, #2068, #1326, #1397.
The solution is very much not ideal, but fixes all math function related
incompatibilities.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3033
This is a collection of fixes for materialized views ahead of adding
support for JOINs.
It is mostly issues with how we assume there is a single table, with a
single delta, but we have to send more than one.
Those are things that are just objectively wrong, so I am sending it
separately to make the JOIN PR smaller.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3009
Currently, when MVCC is enabled, every transaction mode supports
concurrent reads and writes, which makes it hard to adopt for existing
applications that use `BEGIN DEFERRED` or `BEGIN IMMEDIATE`.
Therefore, add support for `BEGIN CONCURRENT` transactions when MVCC is
enabled. The transaction mode allows multiple concurrent read/write
transactions that don't block each other, with conflicts resolved at
commit time. Furthermore, implement the correct semantics for `BEGIN
DEFERRED` and `BEGIN IMMEDIATE` by taking advantage of the pager level
write lock when transaction upgrades to write. This means that now
concurrent MVCC transactions are serialized against the legacy ones when
needed.
The implementation includes:
- Parser support for CONCURRENT keyword in BEGIN statements
- New Concurrent variant in TransactionMode to distinguish from regular
read/write transactions
- MVCC store tracking of exclusive transactions to support IMMEDIATE and
EXCLUSIVE modes alongside CONCURRENT
- Proper transaction state management for all transaction types in MVCC
This enables better concurrency for applications that can handle
optimistic concurrency control, while still supporting traditional
SQLite transaction semantics via IMMEDIATE and EXCLUSIVE modes.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#3021
Our simulator is currently limited to concurrency of one. This
introduces a much less sophisticated DST with focus on finding
concurrency bugs.
Closes#2985
Currently, when MVCC is enabled, every transaction mode supports
concurrent reads and writes, which makes it hard to adopt for existing
applications that use `BEGIN DEFERRED` or `BEGIN IMMEDIATE`.
Therefore, add support for `BEGIN CONCURRENT` transactions when MVCC is
enabled. The transaction mode allows multiple concurrent read/write
transactions that don't block each other, with conflicts resolved at
commit time. Furthermore, implement the correct semantics for `BEGIN
DEFERRED` and `BEGIN IMMEDIATE` by taking advantage of the pager level
write lock when transaction upgrades to write. This means that now
concurrent MVCC transactions are serialized against the legacy ones when
needed.
The implementation includes:
- Parser support for CONCURRENT keyword in BEGIN statements
- New Concurrent variant in TransactionMode to distinguish from regular
read/write transactions
- MVCC store tracking of exclusive transactions to support IMMEDIATE and
EXCLUSIVE modes alongside CONCURRENT
- Proper transaction state management for all transaction types in MVCC
This enables better concurrency for applications that can handle
optimistic concurrency control, while still supporting traditional
SQLite transaction semantics via IMMEDIATE and EXCLUSIVE modes.
A DeltaSet is a collection of Deltas, one per table.
We'll need that for joins. The populate step for now will still generate
a single set. That will be our next step to fix.
Value conversion to float for math functions work in a more strict way
than general numeric conversion. For example, valid prefixes that can be
converted to a integer, like `"44s"` will be converted to `Value::Null`
instead of trying to recover like the math operators.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#3012
If upgrade from read to write transaction fails, don't roll back the
transaction. Instead restore the transaction into its original state,
which allows deferred transactions that have not read anything to
restart automatically.
Fixes#2984
This PR restructure JS packages and also adds support for OPFS for
tursodatabase in browser.
The new structure looks like this:
1. `@tursodatabase/database-common` - contains abstract JS code for
bindings which depends only on `NativeDB` interface and not on the
explicit native bindings
2. `@tursodatabase/database` - contains native bindings for the database
and re-use `core` package
3. `@tursodatabase/database-browser` - contains bindings for browser
(WASM + OPFS)
As OPFS sync API (which is the most performant one in the web) works
only in the web worker - this PR also make few operations async in order
to run them as `napi-rs` AsyncTask. The following operations became
async in `promise.ts` for node and browser: `pragma` / `exec` / `close`.
Also, as few code pathes during initialization are non-async - they
complicates integration of sync constructor in the browser with OPFS.
So, right now - turso support only `connect` method for browser in non-
memory mode.
Closes#2927
hell yeah
concurrency tests passing now woosh
finally write tests passed
Most of the cdc tests are passing yay
autoincremeent draft
remove shared schema code that broke transactions
sequnce table should reset if table is drop
fmt
fmt
fmt
The MakeRecord instruction now accepts an optional affinity_str
parameter that applies column-specific type conversions before creating
records. When provided, the affinity string is applied
character-by-character to each register using the existing
apply_affinity_char() function, matching SQLite's behavior.
Fixes#2040Fixes#2041
On the syscall IO backend, on TPC-H query 12, the _dominating_ part
of the stack trace is trying to construct affinities from a character,
failing, allocating an error&string, and then immediately falling back to
Blob affinity and dropping the error&string.
Since I'm on vacation I won't spend cycles on figuring out why we are passing
an incorrect affinity in `flags.get_affinity()` and instead make this lazy PR
just to improve performance and stop doing silly things :]
This fairly long commit implements persistence for materialized view.
It is hard to split because of all the interdependencies between components,
so it is a one big thing. This commit message will at least try to go into
details about the basic architecture.
Materialized Views as tables
============================
Materialized views are now a normal table - whereas before they were a virtual
table. By making a materialized view a table, we can reuse all the
infrastructure for dealing with tables (cursors, etc).
One of the advantages of doing this is that we can create indexes on view
columns. Later, we should also be able to write those views to separate files
with ATTACH write.
Materialized Views as Zsets
===========================
The contents of the table are a ZSet: rowid, values, weight. Readers will
notice that because of this, the usage of the ZSet data structure dwindles
throughout the codebase. The main difference between our materialized ZSet and
the standard DBSP ZSet, is that obviously ours is backed by a BTree, not a Hash
(since SQLite tables are BTrees)
Aggregator State
================
In DBSP, the aggregator nodes also have state. To store that state, there is a
second table. The table holds all aggregators in the view, and there is one
table per view. That is __turso_internal_dbsp_state_{view_name}. The format of
that table is similar to a ZSet: rowid, serialized_values, weight. We serialize
the values because there will be many aggregators in the table. We can't rely
on a particular format for the values.
The Materialized View Cursor
============================
Reading from a Materialized View essentially means reading from the persisted
ZSet, and enhancing that with data that exists within the transaction.
Transaction data is ephemeral, so we do not materialize this anywhere: we have
a carefully crafted implementation of seek that takes care of merging weights
and stitching the two sets together.
