Closes#1528 .
- Modified `translate_select` so that the caller can define if the
statement is top-level statement or a subquery.
- Refactored `translate_insert` to offload the translation of multi-row
VALUES and SELECT statements to `translate_select`
- I did not try to change much of `populate_column_registers` as I did
not want to break `translate_virtual_table_insert`. Ideally, I would
want to unite this remaining logic folding `populate_column_registers`
into `populate_columns_multiple_rows` and the
`translate_virtual_table_insert` into `translate_insert`. But, I think
this may be best suited for a separate PR.
## TODO
- ~Tests~ - *Done*
- ~Need to emit a temp table when we are selecting and inserting into
the Same Table -
https://github.com/sqlite/sqlite/blob/master/src/insert.c#L1369~ -
*Done*
- Optimization when table have the exact same schema - open an Issue
about it
- Virtual Tables do not benefit yet from this feature - open an Issue
about it
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1566
Before #1570 , we never read anything from the on-disk WAL at startup,
so the `test_wal_frame_count()` test assertion `frame_count == 0`would
always pass, regardless of whether there was anything in the WAL or not
Now that the actual bug is fixed, the current test isn't idempotent
because the `clone_test_db.sh` command does not clear the wal, just the
db file
Closes#1581
- Instead of using a confusing `CheckpointStatus` for many different
things, introduce the following statuses:
* `PagerCacheflushStatus` - cacheflush can result in either:
- the WAL being written to disk and fsynced
- but also a checkpoint to the main DB file, and fsyncing the
main DB file
Reflect this in the type.
* `WalFsyncStatus`
- previously `CheckpointStatus` was also used for this, even
though fsyncing the WAL doesn't checkpoint.
* `CheckpointStatus`/`CheckpointResult` is now used only for actual
checkpointing.
- Rename `HaltState` to `CommitState` (program.halt_state ->
program.commit_state)
* `HaltState`'s only variant was `HaltState::Checkpointing` but it
wasn't necessarily reflective of checkpoint being in progress.
* Now the variants are `CommitState::Ready` and
`CommitState::Committing` and `commit_state` is
non-optional in `ProgramState`
- Make `wal` a non-optional property in `Pager`
* This gets rid of a lot of `if let Some(...)` boilerplate
* For ephemeral indexes, provide a `DummyWAL` implementation that does
nothing.
- Rename `program.halt()` to `program.commit_txn()`
- Add some documentation comments to structs and functions
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#1578
This PR:
- Fix compatibility for variadic functions;
- Adds pragma, transaction, close, and pluck methods;
- Add variables to get, iterate, and all methods
Closes#1576
To whomever may be reviewing this PR,
**[Blacksmith](https://www.blacksmith.sh/) is the fastest way to run
your GitHub Actions.**
## What does this PR change?
This PR has been automatically generated by a team member in your GitHub
organization using Blacksmith's [Migration
Wizard](https://docs.blacksmith.sh/introduction/quickstart), or MigWiz
for short. This PR changes the following:
1. Updates your selected workflows to use Blacksmith's `runs-on:
blacksmith-4vcpu-ubuntu-2204`. Learn more about the [different instances
available to choose from](https://docs.blacksmith.sh/blacksmith-
runners/config).
2. Updates your `actions/cache`, `actions/setup-*` to use Blacksmith's
`useblacksmith/cache`, `useblacksmith/setup-*`. Learn more about
[Blacksmith's cache action](https://docs.blacksmith.sh/blacksmith-
caching/dependencies-actions).
## FAQ
- Is this free? The first 3,000 minutes per month are free.
- Who uses Blacksmith? Clerk, Ashby, VEED, and 300+ others.
- What's the catch? There is none. Merge this thing already.
Closes#1579
- Instead of using a confusing CheckpointStatus for many different things,
introduce the following statuses:
* PagerCacheflushStatus - cacheflush can result in either:
- the WAL being written to disk and fsynced
- but also a checkpoint to the main BD file, and fsyncing the main DB file
Reflect this in the type.
* WalFsyncStatus - previously CheckpointStatus was also used for this, even
though fsyncing the WAL doesn't checkpoint.
* CheckpointStatus/CheckpointResult is now used only for actual checkpointing.
- Rename HaltState to CommitState (program.halt_state -> program.commit_state)
- Make WAL a non-optional property in Pager
* This gets rid of a lot of if let Some(...) boilerplate
* For ephemeral indexes, provide a DummyWAL implementation that does nothing.
- Rename program.halt() to program.commit_txn()
- Add some documentation comments to structs and functions
```sql
limbo> select * from products where id between 1 and 3
UNION ALL
select * from products where id between 2 and 4;
┌───┬─────────┬──────┐
│ 1 │ hat │ 79.0 │
├───┼─────────┼──────┤
│ 2 │ cap │ 82.0 │
├───┼─────────┼──────┤
│ 3 │ shirt │ 18.0 │
├───┼─────────┼──────┤
│ 2 │ cap │ 82.0 │
├───┼─────────┼──────┤
│ 3 │ shirt │ 18.0 │
├───┼─────────┼──────┤
│ 4 │ sweater │ 25.0 │
└───┴─────────┴──────┘
limbo> select * from products where id between 1 and 3
UNION
select * from products where id between 2 and 4;
┌───┬─────────┬──────┐
│ 1 │ hat │ 79.0 │
├───┼─────────┼──────┤
│ 2 │ cap │ 82.0 │
├───┼─────────┼──────┤
│ 3 │ shirt │ 18.0 │
├───┼─────────┼──────┤
│ 4 │ sweater │ 25.0 │
└───┴─────────┴──────┘
limbo>
```
Similarly as UNION ALL (#1541 ), supports LIMIT but not OFFSET or ORDER
BY.
Augments `compound_select_fuzz()` to work with both UNION and UNION ALL
Closes#1545
Closes#1557
Currently our "table id"/"table no"/"table idx" references always use
the direct index of the `TableReference` in the plan, e.g. in
`SelectPlan::table_references`. For example:
```rust
Expr::Column { table: 0, column: 3, .. }
```
refers to the 0'th table in the `table_references` list.
This is a fragile approach because it assumes the table_references list
is stable for the lifetime of the query processing. This has so far been
the case, but there exist certain query transformations, e.g. subquery
unnesting, that may fold new table references from a subquery (which has
its own table ref list) into the table reference list of the parent.
If such a transformation is made, then potentially all of the
Expr::Column references to tables will become invalid. Consider this
example:
```sql
-- Assume tables: users(id, age), orders(user_id, amount)
-- Get total amount spent per user on orders over $100
SELECT u.id, sub.total
FROM users u JOIN
(SELECT user_id, SUM(amount) as total
FROM orders o
WHERE o.amount > 100
GROUP BY o.user_id) sub
WHERE u.id = sub.user_id
-- Before subquery unnesting:
-- Main query table_references: [users, sub]
-- u.id refers to table 0, column 0
-- sub.total refers to table 1, column 1
--
-- Subquery table_references: [orders]
-- o.user_id refers to table 0, column 0
-- o.amount refers to table 0, column 1
--
-- After unnesting and folding subquery tables into main query,
-- the query might look like this:
SELECT u.id, SUM(o.amount) as total
FROM users u JOIN orders o ON u.id = o.user_id
WHERE o.amount > 100
GROUP BY u.id;
-- Main query table_references: [users, orders]
-- u.id refers to table index 0 (correct)
-- o.amount refers to table index 0 (incorrect, should be 1)
-- o.user_id refers to table index 0 (incorrect, should be 1)
```
We could ofc traverse every expression in the subquery and rewrite the
table indexes to be correct, but if we instead use stable identifiers
for each table reference, then all the column references will continue
to be correct.
Hence, this PR introduces a `TableInternalId` used in `TableReference`
as well as `Expr::Column` and `Expr::Rowid` so that this kind of query
transformations can happen with less pain. I used a separate newtype
struct for `TableInternalId` because it made the refactor a lot easier
due to not having to spend time thinking which `usize` is what.
---
Potential follow-up: `join_order` can be removed from `SelectPlan`
because the `table_references` vec can simply be sorted after join
reordering, because the `Expr::Column` references will continue to be
valid.
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#1561
Currently our "table id"/"table no"/"table idx" references always
use the direct index of the `TableReference` in the plan, e.g. in
`SelectPlan::table_references`. For example:
```rust
Expr::Column { table: 0, column: 3, .. }
```
refers to the 0'th table in the `table_references` list.
This is a fragile approach because it assumes the table_references
list is stable for the lifetime of the query processing. This has so
far been the case, but there exist certain query transformations,
e.g. subquery unnesting, that may fold new table references from
a subquery (which has its own table ref list) into the table reference
list of the parent.
If such a transformation is made, then potentially all of the Expr::Column
references to tables will become invalid. Consider this example:
```sql
-- Assume tables: users(id, age), orders(user_id, amount)
-- Get total amount spent per user on orders over $100
SELECT u.id, sub.total
FROM users u JOIN
(SELECT user_id, SUM(amount) as total
FROM orders o
WHERE o.amount > 100
GROUP BY o.user_id) sub
WHERE u.id = sub.user_id
-- Before subquery unnesting:
-- Main query table_references: [users, sub]
-- u.id refers to table 0, column 0
-- sub.total refers to table 1, column 1
--
-- Subquery table_references: [orders]
-- o.user_id refers to table 0, column 0
-- o.amount refers to table 0, column 1
--
-- After unnesting and folding subquery tables into main query,
-- the query might look like this:
SELECT u.id, SUM(o.amount) as total
FROM users u JOIN orders o ON u.id = o.user_id
WHERE o.amount > 100
GROUP BY u.id;
-- Main query table_references: [users, orders]
-- u.id refers to table index 0 (correct)
-- o.amount refers to table index 0 (incorrect, should be 1)
-- o.user_id refers to table index 0 (incorrect, should be 1)
```
We could ofc traverse every expression in the subquery and rewrite
the table indexes to be correct, but if we instead use stable identifiers
for each table reference, then all the column references will continue
to be correct.
Hence, this PR introduces a `TableInternalId` used in `TableReference`
as well as `Expr::Column` and `Expr::Rowid` so that this kind of query
transformations can happen with less pain.
Re-Opening #1076 because it had bit-rotted to a point of no return.
However it has improved. Now with Weak references and no incrementing Rc
strong counts.
This also includes a better test extension that returns info about the
other tables in the schema.
(theme doesn't show rows column)
Closes#1366
Fixes#1567
Probably also fixes#1485
Currently we are simply unable to read any WAL frames from disk once a
fresh process w/ Limbo is opened, since we never try to read anything
from disk unless we already have it in our in-memory frame cache.
This commit implements a crude way of reading entire WAL into memory as
a single buffer and reconstructing the frame cache.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#1570
Shared cache requires more locking mechasnisms. We still have multi
threading issues not related to shared cache so it is wise to first fix
those and then once they are fixed, we can incrementally add shared
cache back with locking in place.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1568
