Attempt to improve performance of `SELECT COUNT(*) FROM t WHERE 1 = 1`
(redundant filter added to force sqlite and tursodb to iterate over rows
instead of applying count optimization).
Before that PR turso with hot cache runs for ~440ms.
After that PR turso with hot cache runs for ~200ms.
This PR eliminates small things on a hot path and also introduce
separate flow for `get_next_record` which just increment cell idx if its
safe to do that - without much additional overhead.
Also, this PR eliminates RefCell/Cell from PageStack because there were
completely unnecessary.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2887
This adds the [`json_each`](https://sqlite.org/json1.html#the_json_each_
and_json_tree_table_valued_functions) TVF. Only the 1-arg version is
supported for now.
As suggested in the comments on this PR, I've also extended the virtual
table system to support internal TVF's, as opposed to extensions and
pragma TVF's.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2691
This PR replaces the existing LRU cache implementation with the sieve
algo, providing better performance and simpler code.
Same intrusive linked-list without pointer chasing and no per-
page/insert heap allocations.
Benchmarks look really good so far, although some did show some
regression (I think it's more likely that they were flakey and its not
representative on either end, but I do believe this to be considerably
more performant)
Best numbers I have seen yet on Mobibench locally, with 1473 tx/s.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2885
On the syscall IO backend, on TPC-H query 12, the _dominating_ part of
the stack trace is trying to construct affinities from a character,
failing, allocating an error&string, and then immediately falling back
to Blob affinity and dropping the error&string.
Since I'm on vacation I won't spend cycles on figuring out why we are
passing an incorrect affinity in `flags.get_affinity()` and instead make
this lazy PR just to improve performance and stop doing silly things :]
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#2941
On the syscall IO backend, on TPC-H query 12, the _dominating_ part
of the stack trace is trying to construct affinities from a character,
failing, allocating an error&string, and then immediately falling back to
Blob affinity and dropping the error&string.
Since I'm on vacation I won't spend cycles on figuring out why we are passing
an incorrect affinity in `flags.get_affinity()` and instead make this lazy PR
just to improve performance and stop doing silly things :]
Add handling malformed inputs to function `read_varint` and test cases.
```
# 9 byte truncated to 8
read_varint(&[0x81, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80])
before -> panic index out of bounds: the len is 8 but the index is 8
after -> LimboError
# bits set without end
read_varint(&[0x80; 9])
before -> Ok((128, 9))
after -> LimboError
```
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#2904
fix#2937 , we dont need to check comment in cli because parser already
handled it (i guest).
How to reproduce:
```sh
turso> --whatCREATE TABLE users (
id INT PRIMARY KEY,
first_name VARCHAR(50),
age INT
);
```
Closes#2938
This fairly long commit implements persistence for materialized view. It
is hard to split because of all the interdependencies between
components, so it is a one big thing. This commit message will at least
try to go into details about the basic architecture.
Materialized Views as tables
============================
Materialized views are now a normal table - whereas before they were a
virtual table. By making a materialized view a table, we can reuse all
the infrastructure for dealing with tables (cursors, etc).
One of the advantages of doing this is that we can create indexes on
view columns. Later, we should also be able to write those views to
separate files with ATTACH write.
Materialized Views as Zsets
===========================
The contents of the table are a ZSet: rowid, values, weight. Readers
will notice that because of this, the usage of the ZSet data structure
dwindles throughout the codebase. The main difference between our
materialized ZSet and the standard DBSP ZSet, is that obviously ours is
backed by a BTree, not a Hash (since SQLite tables are BTrees)
Aggregator State
================
In DBSP, the aggregator nodes also have state. To store that state,
there is a second table. The table holds all aggregators in the view,
and there is one table per view. That is
__turso_internal_dbsp_state_{view_name}. The format of that table is
similar to a ZSet: rowid, serialized_values, weight. We serialize the
values because there will be many aggregators in the table. We can't
rely on a particular format for the values.
The Materialized View Cursor
============================
Reading from a Materialized View essentially means reading from the
persisted ZSet, and enhancing that with data that exists within the
transaction. Transaction data is ephemeral, so we do not materialize
this anywhere: we have a carefully crafted implementation of seek that
takes care of merging weights and stitching the two sets together.
Closes#2921
This fairly long commit implements persistence for materialized view.
It is hard to split because of all the interdependencies between components,
so it is a one big thing. This commit message will at least try to go into
details about the basic architecture.
Materialized Views as tables
============================
Materialized views are now a normal table - whereas before they were a virtual
table. By making a materialized view a table, we can reuse all the
infrastructure for dealing with tables (cursors, etc).
One of the advantages of doing this is that we can create indexes on view
columns. Later, we should also be able to write those views to separate files
with ATTACH write.
Materialized Views as Zsets
===========================
The contents of the table are a ZSet: rowid, values, weight. Readers will
notice that because of this, the usage of the ZSet data structure dwindles
throughout the codebase. The main difference between our materialized ZSet and
the standard DBSP ZSet, is that obviously ours is backed by a BTree, not a Hash
(since SQLite tables are BTrees)
Aggregator State
================
In DBSP, the aggregator nodes also have state. To store that state, there is a
second table. The table holds all aggregators in the view, and there is one
table per view. That is __turso_internal_dbsp_state_{view_name}. The format of
that table is similar to a ZSet: rowid, serialized_values, weight. We serialize
the values because there will be many aggregators in the table. We can't rely
on a particular format for the values.
The Materialized View Cursor
============================
Reading from a Materialized View essentially means reading from the persisted
ZSet, and enhancing that with data that exists within the transaction.
Transaction data is ephemeral, so we do not materialize this anywhere: we have
a carefully crafted implementation of seek that takes care of merging weights
and stitching the two sets together.
The CI is sometimes failing with the following error:
```thread '<unnamed>' panicked at sqlite3/src/lib.rs:156:37:
called `Result::unwrap()` on an `Err` value: InternalError("WAL already enabled")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at library/core/src/panicking.rs:226:5:
panic in a function that cannot unwind
stack backtrace:
0: 0x7ff612e11502 - std::backtrace_rs::backtrace::libunwind::trace::h74680e970b6e0712
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
1: 0x7ff612e11502 - std::backtrace_rs::backtrace::trace_unsynchronized::ha3bf590e3565a312
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
2: 0x7ff612e11502 - std::sys::backtrace::_print_fmt::hcf16024cbdd6c458
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/sys/backtrace.rs:66:9
3: 0x7ff612e11502 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h46a716bba2450163
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/sys/backtrace.rs:39:26
4: 0x7ff612e338a3 - core::fmt::rt::Argument::fmt::ha695e732309707b7
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/fmt/rt.rs:181:76
5: 0x7ff612e338a3 - core::fmt::write::h275e5980d7008551
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/fmt/mod.rs:1446:25
6: 0x7ff612e0f003 - std::io::default_write_fmt::hdc4119be3eb77042
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/io/mod.rs:639:11
7: 0x7ff612e0f003 - std::io::Write::write_fmt::h561a66a0340b6995
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/io/mod.rs:1914:13
8: 0x7ff612e11352 - std::sys::backtrace::BacktraceLock::print::hafb9d5969adc39a0
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/sys/backtrace.rs:42:9
9: 0x7ff612e12842 - std::panicking::default_hook::{{closure}}::hae2e97a5c4b2b777
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:300:22
10: 0x7ff612e12645 - std::panicking::default_hook::h3db1b505cfc4eb79
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:327:9
11: 0x7ff612e131e2 - std::panicking::rust_panic_with_hook::h409da73ddef13937
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:833:13
12: 0x7ff612e12f56 - std::panicking::begin_panic_handler::{{closure}}::h159b61b27f96a9c2
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:699:13
13: 0x7ff612e119f9 - std::sys::backtrace::__rust_end_short_backtrace::h5b56844d75e766fc
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/sys/backtrace.rs:168:18
14: 0x7ff612e12c1d - __rustc[4794b31dd7191200]::rust_begin_unwind
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/std/src/panicking.rs:697:5
15: 0x7ff611f56f9d - core::panicking::panic_nounwind_fmt::runtime::h4c94eb695becba00
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:117:22
16: 0x7ff611f56f9d - core::panicking::panic_nounwind_fmt::hc3cf3432011a3c3f
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/intrinsics/mod.rs:3196:9
17: 0x7ff611f57032 - core::panicking::panic_nounwind::h0c59dc9f7f043ead
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:226:5
18: 0x7ff611f57191 - core::panicking::panic_cannot_unwind::hb8732afd89555502
at /rustc/6b00bc3880198600130e1cf62b8f8a93494488cc/library/core/src/panicking.rs:331:5
19: 0x7ff611f585ee - sqlite3_open
at /home/runner/_work/turso/turso/sqlite3/src/lib.rs:127:1
20: 0x55f88e591f2c - compat::tests::test_wal_checkpoint_v2::hc70b8ddc1bc8d78d
at /home/runner/_work/turso/turso/sqlite3/tests/compat/mod.rs:188:17
```
As Rust integration tests run in parallel, they cannot use the same
tests files. However, as it turns out, one test does not even need a
file, and the other two are redundant so we can remote them.
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#2936
They're both using the same database file which is wrong (tests run in
parallel). But more importantly, they test almost nothing, and we have a
better checkpoint test already.
