This PR improves sync and database bindings for browser
List of changes:
- For node and browser database now run on main thread and only IO work
offloaded to the worker (web worker in browser)
- Simple locks are implemented for database access externally in order
to guard access to the same connection (when request is executed async -
main thread can try to start another request concurrently)
- parking_lot in the Wal replaced by spin-wait (by invoking
`parking_lot.try_read/try_write`) for WASM target because browser can't
park main thread
- js sync reworked in order to support few engine options
(`longPollTimeoutMs`) and introduce external locking which properly
guards concurrent access of sync methods
Closes#3218
Previously we were rewriting/traversing the AST in a couple different
places, each of these added kinda ad-hoc as we needed them. This
attempts to do the binding of column references as well as the rewriting
of anonymous `Expr::Variable` -> `__param_N` that we use to maintain the
order of bound variables, also normalizes the Qualified Name's.
Also we previously weren't accepting Variable (or at least they wouldn't
work) in places like `LIMIT ? OFFSET ?`, which this PR adds.
I kinda want to keep refactoring translation a bit, and try to break
plan building up into more easy-to-digest chunks.. but I will resist the
urge right now as it's definitely not high priority pre-beta
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#3210
I don't want to even think about the complexity involved in making sure
that materialized views are still sane after the base table(s) are
altered.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3223
This PR introduces the final major operator: the JOIN operator.
Many things need to be fixed before we can properly support them, and we
handle those. In particular, JOINs always generate qualified column
statements, but we were not handling them correctly at all in the
operators. Not a problem for linear circuits, but fatal for JOINs.
The operator.rs file also becomes incredibly complex with not one, but
two stateful operator. So it is now broken apart.
Closes#3207
I keep getting these warnings and it slightly annoys me:
```rust
warning: unused variable: `overflow_cell_count_before`
--> core/storage/btree.rs:3634:29
|
3634 | let overflow_cell_count_before = parent_contents.overflow_cells.len();
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_overflow_cell_count_before`
|
= note: `#[warn(unused_variables)]` on by default
warning: unused variable: `overflow_cell_count_after`
--> core/storage/btree.rs:3641:29
|
3641 | let overflow_cell_count_after = parent_contents.overflow_cells.len();
| ^^^^^^^^^^^^^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_overflow_cell_count_after`
```
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#3219
- now, most of the work is happening on the main thread
- for database in browser, we still have dedicated WebWorker - but it is used only for OPFS access and only for that
- for syn in browser we still offload sync operations to the WebWorker
We currently only support column / literal comparisons in the filter
operator. But with JOINs, comparisons are usually against two columns.
Do the work to support it.
Unlike the other operators, project works just fine with ambiguous
columsn, because it works with compiled expressions. We don't need to
patch it, but let's make sure it keeps working by writing a test.
We already did similarly for the AggregateOperator: for joins
you can have the same column name in many tables. And passing schema
information to the operator is a layering violation (the operator may be
operating on the result of a previous node, and at that point there is
no more "schema"). Therefore we pass indexes into the column set the
operator has.
The FilterOperator has a complication: we are using it to generate the
SQL for the populate statement, and that needs column names. However,
we should *not* be using the FilterOperator for that, and that is a
relic from the time where we had operator information directly inside
the IncrementalView.
To enable moving the FilterOperator to index-based, we rework that code.
For joins, we'll need to populate many tables anyway, so we take the
time to do that work here.
This PR improves the DBSP circuit so that it handles the JOIN operator.
The JOIN operator exposes a weakness of our current model: we usually
pass a list of columns between operators, and find the right column by
name when needed.
But with JOINs, many tables can have the same columns. The operators
will then find the wrong column (same name, different table), and
produce incorrect results.
To fix this, we must do two things:
1) Change the Logical Plan. It needs to track table provenance.
2) Fix the aggregators: it needs to operate on indexes, not names.
For the aggregators, note that table provenance is the wrong
abstraction. The aggregator is likely working with a logical table that
is the result of previous nodes in the circuit. So we just need to be
able to tell it which index in the column array it should use.
The operator.rs file was so huge, that we didn't even notice there was a
test block in the middle of the file that was testing things that were
long moved to dbsp.rs (the HashableRow). Move the tests there now.
