Closes#2618 (sufficiently for now, IMO)
Prevents simulator from creating malformed queries which we don't notice
(ref e.g. #2609 , #2611, #2616)
Closes#2624
We were not generating table_info for views. This PR fixes it. We were
so far storing columns as strings with just their names - since this is
all we needed - but we will move now to store Columns. We need to
convert the names to Column anyway for table_info to work.
Closes#2625
We have halt and op_halt, doing essentially the same thing.
This PR unifies them. There is a minor difference between them now in
the way halt() handles auto-commit. My current understanding of the code
is that what we have in halt *is a bug*, which is already one bad
consequence of the duplication.
Closes#2631
Currently during `Restart|Truncate` modes we properly restart the WAL
header in the `WalFileShared`, but keep a stale one on the current
`WalFile` instance.
Closes#2627
We have halt and op_halt, doing essentially the same thing.
This PR unifies them. There is a minor difference between them now in
the way halt() handles auto-commit. My current understanding of the code
is that what we have in halt *is a bug*, which is already one bad
consequence of the duplication.
We were not generating table_info for views. This PR fixes it. We were
so far storing columns as strings with just their names - since this is
all we needed - but we will move now to store Columns. We need to
convert the names to Column anyway for table_info to work.
Also fixes an issue where since the old Database wasn't `Drop`d, it
caused an issue with the static Registry ™️ that had broken
`--doublecheck` mode.
Closes#2623
This continues the work for `printf` support from from tthe issue
https://github.com/tursodatabase/turso/issues/885.
This PR adds support for `%i` operands in the printf function. An
example of a command that works now and didn't before:
```sql
SELECT PRINTF("Decimals: %d %i", 200, 300);
```
This is Just an initial very simple contribution to get my feet wet and
learn the basics about the project and how contributions here work. I
intend to do something more meaningful in next contributions. Probably
will try to finish support for missing `printf` functionality also.
Closes#2615
This PR switches from usage of WAL push to SQL over http calls.
As we have sequence of logical changes from CDC - we can just send SQL
statements to the remote. This will give us benefits of more smooth
concurrent usage of sync - because WAL push is very unfriendly in case
of concurrent usages.
Closes#2617
There is a distinction between tests that verify extension-specific
behavior and those that verify interactions between the database engine
and extensions. Previously, both types of tests were kept in
`extensions.py`. With this new framework, we can extract the latter type
of tests from `extensions.py` into TCL. This cleans up `extensions.py`
and enables compatibility testing with SQLite at no extra cost.
I’m currently working on supporting outer joins involving TVFs and
planning to add more tests that exercise the database’s handling of
virtual tables, so I decided to do this refactoring first.
In the future, we may consider moving extension-specific tests to TCL as
well, especially those that have counterparts in SQLite or sqlean.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2556
## 1. select equal number of columns per compound subselect in simulator
#2609 fixed compound selects incorrectly, introducing another bug: now
the sim can SELECT a different number of columns per subselect, which is
illegal. this PR forces the sim to select the same number of cols per
subselect.
Depends on #2614 , otherwise the sim constantly fails whenever it
decides to do a compound select with DISTINCT.
Closes#2611
## 2. introduce TableContext for the simulator to properly generate
predicates for Joins
The simulator has a construct called `JoinTable` which is really a
`JoinContext` that includes all the columns from the so-far joined
tables, so that the next table to be joined can refer to columns from
any of those tables. @pedrocarlo 's commit fixes an issue where
`JoinTable` would incorrectly generate predicates with empty table
names. Related: #2618Closes#2616
Closes#2612
Two commits:
## Simplify ORDER BY sorter column remapping
In case an ORDER BY column exactly matches a result column in the
SELECT,
the insertion of the result column into the ORDER BY sorter can be
skipped
because it's already necessarily inserted as a sorting column.
For this reason we have a mapping to know what index a given result
column
has in the order by sorter.
This commit makes that mapping much simpler.
## Fix DISTINCT with ORDER BY
We had a bug where we were checking for duplicates in the DISTINCT
index based on both the result column count plus any ORDER BY columns
not present in the DISTINCT clause.
This is wrong, so fix it by only using the result columns for the
dedupe check.
Closes#2614
These tests verify interactions between the database engine and TVFs.
They happen to use generate_series, but they are not intended to test
the behavior of any specific extension. Tests that verify generate_series
specific behavior remain in extensions.py.
There is a distinction between tests that verify extension-specific
behavior and tests that verify interactions between the database engine
and extensions. Previously, both types of tests were kept in extensions.py.
With this new framework, we can extract the latter type of tests from
extensions.py into TCL. This cleans up extensions.py and provides
compatibility testing with SQLite at no extra cost.
To demonstrate the framework’s usage, tests verifying the handling of
virtual tables were extracted to TCL.
In the future, we may consider moving extension-specific tests to TCL as
well, especially those that have counterparts in SQLite or sqlean.
We had a bug where we were checking for duplicates in the DISTINCT
index based on both the result column count plus any ORDER BY columns
not present in the DISTINCT clause.
This is wrong, so fix it by only using the result columns for the
dedupe check.
In case an ORDER BY column exactly matches a result column in the SELECT,
the insertion of the result column into the ORDER BY sorter can be skipped
because it's already necessarily inserted as a sorting column.
For this reason we have a mapping to know what index a given result column
has in the order by sorter.
This commit makes that mapping much simpler.
Set in-memory WAL information according to last commited frame
- before that pages_in_frames, frame_cache and last_checksum was set to
the latest written (not commited!) frame in the WAL found on disk
Simple example to reproduce the data-loss:
```sh
n=$RANDOM
./target/debug/tursodb data-loss-$n.db 'create table t(x)';
strace -o /dev/null -e fault=pwrite64:error=EIO:when=5 ./target/debug/tursodb data-loss-$n.db 'insert into t values (randomblob(4 * 4096));'
./target/debug/tursodb data-loss-$n.db 'insert into t values (1);'
./target/debug/tursodb data-loss-$n.db 'select x from t;'
```
The main bug which contributes to the data-loss was update of
`last_checksum` to the value from last frame read from WAL while
actually DB must use checksum of last **commited** frame. Due to this
bug, `insert` from third operation used checksum from last **written**
frame, but continued to append wal from the position after last
**committed** frame. So, fourth operation read WAL, detected checksum
mismatch and dropped all frames written by third operation.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#2613
