This PR adds a python script that uses the `TestLimboShell` setup to run
some semi naive benchmarks/comparisons against `io_uring` and `syscall`
IO back-ends.
### Usage:
```sh
make bench-vfs SQL="insert into products (name, price) values ('testing', randomblob(1024*4));" N=50
```
The script will execute the given `SQL` `N` times with each back-end,
get the average/mean and display them.
😬
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1377
I was reading through the `translate_expr` function and `COMPAT.md` to
see what was not implemented yet. I saw that `Expr::Between` was marked
as a `todo!` so I set trying to implement it only to find that it was
being rewritten in the optimizer haha. This PR just adjusts the docs and
add an `unreachable` in the appropriate locations.
Closes#1378
This PR adds the statement.columns() function, inspired from Rusqlite: h
ttps://docs.rs/rusqlite/latest/rusqlite/struct.Statement.html#method.col
umns
Note that the rusqlite documentation says
> If associated DB schema can be altered concurrently, you should make
sure that current statement has already been stepped once before calling
this method.
Do we have this requirement as well?
The first commit is just the rust binding. The second commit implements
the column name for the rowid column.
Closes#1376
The previous version of `julian_day-converter` had precision issues,
potentially causing loss of precision when converting between
`julianday` and `datetime`
Reviewed-by: Diego Reis (@diegoreis42)
Closes#1344
This PR is based on #1357 and further improves performance:
```sql
limbo> select l_orderkey, 3 as revenue, o_orderdate, o_shippriority from lineitem, orders, customer where c_mktsegment = 'FURNITURE' and c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate < cast('1995-03-29' as datetime) and l_shipdate > cast('1995-03-29' as datetime);
┌────────────┬─────────┬─────────────┬────────────────┐
│ l_orderkey │ revenue │ o_orderdate │ o_shippriority │
├────────────┼─────────┼─────────────┼────────────────┤
└────────────┴─────────┴─────────────┴────────────────┘
Command stats:
----------------------------
total: 3.728050958 s (this includes parsing/coloring of cli app)
```
Reviewed-by: Preston Thorpe (@PThorpe92)
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#1358
In left joins, even if the join condition is not matched, the system
must emit a row for every row of the outer table:
```
-- this must return t1.count() rows, with NULLs for all columns of t2
SELECT * FROM t1 LEFT JOIN t2 ON FALSE;
```
To achieve this, we set a "null flag" on the right table cursor which
tells our VDBE to emit NULLs for any columns of that cursor until the
flag is cleared.
Our logic for clearing the null flag was to do it in Next/Prev. However,
this is problematic for a few reasons:
- If the inner table of the left join is using SeekRowid, then Next/Prev
is never called on its cursor, so the null flag doesn't get cleared.
- If the inner table of the left join is using a non-covering index
seek, i.e. it iterates its rows using an index, but seeks to the main
table to fetch data, then Next/Prev is never called on the main table,
and the main table's null flag doesn't get cleared.
What this results in is NULL values incorrectly being emitted for the
inner table after the first correct NULL row, since the null flag is
correctly set to true, but never cleared.
This PR fixes the issue by clearing the null flag whenever seek() is
invoked on the cursor. Hence, the null flag is now cleared on:
- next()
- prev()
- seek()
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#1364
In left joins, even if the join condition is not matched, the system
must emit a row for every row of the outer table:
-- this must return t1.count() rows, with NULLs for all columns of t2
SELECT * FROM t1 LEFT JOIN t2 ON FALSE;
Our logic for clearing the null flag was to do it in Next/Prev. However,
this is problematic for a few reasons:
- If the inner table of the left join is using SeekRowid, then Next/Prev
is never called on its cursor, so the null flag doesn't get cleared.
- If the inner table of the left join is using a non-covering index seek,
i.e. it iterates its rows using an index, but seeks to the main table
to fetch data, then Next/Prev is never called on the main table, and the
main table's null flag doesn't get cleared.
What this results in is NULL values incorrectly being emitted for the
inner table after the first correct NULL row, since the null flag is
correctly set to true, but never cleared.
This PR fixes the issue by clearing the null flag whenever seek() is
invoked on the cursor. Hence, the null flag is now cleared on:
- next()
- prev()
- seek()
Implements binary search to find the correct cell within a page,
specialized for table btrees only due to lack of energy at 8:30 PM
---
I used a [1GB TPC-H database](https://github.com/lovasoa/TPCH-
sqlite/releases/download/v1.0/TPC-H.db) for benchmarking and ran this
query which does a lot of seeks:
before
```sql
limbo> .timer on
limbo> select
l_orderkey,
3 as revenue,
o_orderdate,
o_shippriority
from
lineitem,
orders,
customer
where
c_mktsegment = 'FURNITURE'
and c_custkey = o_custkey
and l_orderkey = o_orderkey
and o_orderdate < cast('1995-03-29' as datetime)
and l_shipdate > cast('1995-03-29' as datetime);
┌────────────┬─────────┬─────────────┬────────────────┐
│ l_orderkey │ revenue │ o_orderdate │ o_shippriority │
├────────────┼─────────┼─────────────┼────────────────┤
└────────────┴─────────┴─────────────┴────────────────┘
Command stats:
----------------------------
total: 16.267797375 s (this includes parsing/coloring of cli app)
```
after
```sql
limbo> .timer on
limbo> select
l_orderkey,
3 as revenue,
o_orderdate,
o_shippriority
from
lineitem,
orders,
customer
where
c_mktsegment = 'FURNITURE'
and c_custkey = o_custkey
and l_orderkey = o_orderkey
and o_orderdate < cast('1995-03-29' as datetime)
and l_shipdate > cast('1995-03-29' as datetime);
┌────────────┬─────────┬─────────────┬────────────────┐
│ l_orderkey │ revenue │ o_orderdate │ o_shippriority │
├────────────┼─────────┼─────────────┼────────────────┤
└────────────┴─────────┴─────────────┴────────────────┘
Command stats:
----------------------------
total: 5.20604125 s (this includes parsing/coloring of cli app)
```
BTW sqlite completes this in 600 milliseconds so there's still a lot of
fuckiness somewhere.
---
UPDATE: refactored table btree seek (on leaf pages) to use binary search
too. I also updated the above numbers so that I ran each a few times and
took the lowest time i got for each. This is after binsearch on leaf
pages too:
```sql
limbo> select l_orderkey, 3 as revenue, o_orderdate, o_shippriority from lineitem, orders, customer where c_mktsegment = 'FURNITURE' and c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate < cast('1995-03-29' as datetime) and l_shipdate > cast('1995-03-29' as datetime);
┌────────────┬─────────┬─────────────┬────────────────┐
│ l_orderkey │ revenue │ o_orderdate │ o_shippriority │
├────────────┼─────────┼─────────────┼────────────────┤
└────────────┴─────────┴─────────────┴────────────────┘
Command stats:
----------------------------
total: 4.529645958 s (this includes parsing/coloring of cli app)
```
Closes#1357
