After reading #1123, I wanted to see what optimizations I could do.
Sqlite optimizes `count` aggregation for the following case: `SELECT
count() FROM <tbl>`. This is so widely used, that they made an
optimization just for it in the form of the `COUNT` opcode.
This PR thus implements this optimization by creating the `COUNT`
opcode, and checking in the select emitter if we the query is a Simple
Count Query. If it is, we just emit the Opcode instead of going through
a Rewind loop, saving on execution time.
The screenshots below show a huge decrease in execution time.
- **Main**
<img width="383" alt="image" src="https://github.com/user-
attachments/assets/99a9dec4-e7c5-41db-ba67-4eafa80dd2e6" />
- **Count Optimization**
<img width="435" alt="image" src="https://github.com/user-
attachments/assets/e93b3233-92e6-4736-aa60-b52b2477179f" />
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1443
closes#1449
A quick `explain` will show the root of the problem:
vs `sqlite3`
We process the columns in the order they need to be to create the
record, and although we store the index of the value meant to be
inserted into that column based on the order they are bound, we still
lose the original ordering.
## EDIT: Fixed
### Multi-row insert:
## Solution:
We just needed to traverse the insert values beforehand and create an
array of N integers each representing the index of each
`ast::Expr::Variable`: then later on we can lookup the `value_index` in
that array, and grab the index of matching integer to get the
parameters index to be bound to the relevant variable.
(yes I know that sounds like a leetcode problem)
Thanks to @jnesss for the write-up for this bug
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1459
This commit adds suport for DROP INDEX.
Bytecode produced by this commit differs from SQLITE's bytecode, main
reason we don't do autovacuum or repacking of pages like SQLITE does.
Closes#1280Closes#1444
## Changes
Since limbo now supports UPDATE and DELETE, I'm planning to implement
the rest of the features for integrating JDBC and limbo.
Since limbo's [total_changes() function seems to work differently in
compared to the sqlite's](https://discord.com/channels/12586588262579610
20/1321869557459058778/1368830400985563176), let's remove the
`@Disabled` annotation from
`execute_update_should_return_number_of_updated_elements` test after
the issue is handled.
## Reference
https://github.com/tursodatabase/limbo/issues/615Closes#1451
This pull request includes a small change to the
`bindings/wasm/package.json` file. The change adds a `types` field
pointing to the TypeScript declaration file for the package (`[bindings/
wasm/package.jsonR15](diffhunk://#diff-
1f41234b939ba148924b9bbfedd557853ebcd22351a9c300e568ce7af0291babR15)`).
Closes#1460
This PR implements basic transaction support in the Limbo Go adapter by
adding the required methods to fulfill the `driver.Tx` interface.
## Changes
- Add `Begin()` method to `limboConn` to start a transaction
- Add `BeginTx()` method with context support and proper handling of
transaction options
- Implement `Commit()` method to commit transaction changes
- Implement `Rollback()` method with appropriate error handling
- Add transaction tests
## Implementation Details
- Uses the standard SQLite transaction commands (BEGIN, COMMIT,
ROLLBACK)
- Follows the same pattern as other SQL operations in the adapter
(prepare-execute-close)
- Maintains consistent locking and error handling patterns
## Limitations
- Currently, ROLLBACK operations will return an error as they're not yet
fully supported in the underlying Limbo implementation
- Only the default isolation level is supported; all other isolation
levels return `driver.ErrSkip`
- Read-only transactions are not supported and return `driver.ErrSkip`
## Testing
- Added basic transaction tests that verify BEGIN and COMMIT operations
- Adjusted tests to work with the current Limbo implementation
capabilities
These transaction methods enable the Go adapter to be used in
applications that require transaction support while providing clear
error messages when unsupported features are requested. I'll add to it
when Limbo supports ROLLBACK and/or additional isolation levels.
Closes#1435
Probably this is larger than it should, but this PR:
- Enhance error handling (instead of `unwraps`, napi's `GenericFailure`
with custom message)
- Add missing methods and properties for Database and Statements to turn
explicit what should be done
- Implement `all()` and `run()`* methods for `Statement`
- Bump napi version to the latest stable
- Implement `DatabaseStorage::sync()` for DatabaseFile
- Add conversion from js values to limbo values
\* `run()` isn't 100% compatible yet for two reasons:
1. The function API isn't variadic -- I chatted with napi's maintainer
in Discord and he said that this isn't possible using only napi (in this
version), and we could normalize "[...] parameters at the JavaScript
layer before passing them to Rust". Something similar to [this](https://
github.com/rolldown/rolldown/blob/main/packages/rolldown/src/utils/bindi
ngify-input-options.ts), which I plan to do in another PR.
2. better-sqlite version returns a result object, which isn't currently
supported by core (AFAIK), I also plan to do this in another PR.
Closes#1464
PR #1442 added support for using time.Time as query parameters.
Scanning time.Time values from query results still fails:
`sql: Scan error on column index 4, name "mod_time": unsupported Scan,
storing driver.Value type string into type *time.Time`
This change modifies the `toGoValue` function to detect RFC3339
formatted datetime strings and convert them back to time.Time objects
when reading from the database. This provides complete roundtrip support
for time.Time values.
Also added boolean support by converting Go bool values to integers (0
for false, 1 for true) when binding parameters, following SQLite's
convention for representing boolean values.
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#1465
Go 1.23.9 introduced a change with to it's linker that caused a
`duplicate symbol` error with purego 1.82
this is the recommended fix per
https://github.com/golang/go/issues/73617Closes#1466
Right now we have the following problem with GROUP BY:
- it always allocates a sorter and sorts the input rows, even when the
rows are already sorted in the right order
This PR is a refactor supporting a future PR that introduces a new
version of the optimizer which does 1. join reordering and 2. sorting
elimination based on plan cost. The PR splits GROUP BY into multiple
subsections:
1. Initializing the sorter, if needed
2. Reading rows from the sorter, if needed
3. Doing the actual grouping (this is done regardless of whether sorting
is needed)
4. Emitting rows during grouping in a subroutine (this is done
regardless of whether sorting is needed)
For example, you might currently have the following pseudo-bytecode for
GROUP BY:
```
SorterOpen (groupby_sorter)
OpenRead (users)
Rewind (users)
<read columns from users>
SorterInsert (groupby_sorter)
Next (users)
SorterSort (groupby_sorter)
<do grouping>
SorterNext (groupby_sorter)
ResultRow
```
This PR allows us to do the following in cases where the rows are
already sorted:
```
OpenRead (users)
Rewind (users)
<read columns from users>
<do grouping>
Next (users)
ResultRow
```
---
In fact this is where the vast majority of the changes in this PR come
from -- eliminating the implied assumption that sorting for GROUP BY is
always required. The PR does not change current behavior, i.e. sorting
is always done for GROUP BY, but it adds the _ability_ to not do sorting
if the planner so decides.
The most important changes to understand are these:
```rust
/// Enum representing the source for the rows processed during a GROUP BY.
/// In case sorting is needed (which is most of the time), the variant
/// [GroupByRowSource::Sorter] encodes the necessary information about that
/// sorter.
///
/// In case where the rows are already ordered, for example:
/// "SELECT indexed_col, count(1) FROM t GROUP BY indexed_col"
/// the rows are processed directly in the order they arrive from
/// the main query loop.
#[derive(Debug)]
pub enum GroupByRowSource {
Sorter {
/// Cursor opened for the pseudo table that GROUP BY reads rows from.
pseudo_cursor: usize,
/// The sorter opened for ensuring the rows are in GROUP BY order.
sort_cursor: usize,
/// Register holding the key used for sorting in the Sorter
reg_sorter_key: usize,
/// Number of columns in the GROUP BY sorter
sorter_column_count: usize,
/// In case some result columns of the SELECT query are equivalent to GROUP BY members,
/// this mapping encodes their position.
column_register_mapping: Vec<Option<usize>>,
},
MainLoop {
/// If GROUP BY rows are read directly in the main loop, start_reg is the first register
/// holding the value of a relevant column.
start_reg_src: usize,
/// The grouping columns for a group that is not yet finalized must be placed in new registers,
/// so that they don't get overwritten by the next group's data.
/// This is because the emission of a group that is "done" is made after a comparison between the "current" and "next" grouping
/// columns returns nonequal. If we don't store the "current" group in a separate set of registers, the "next" group's data will
/// overwrite the "current" group's columns and the wrong grouping column values will be emitted.
/// Aggregation results do not require new registers as they are not at risk of being overwritten before a given group
/// is processed.
start_reg_dest: usize,
},
}
/// Enum representing the source of the aggregate function arguments
/// emitted for a group by aggregation.
/// In the common case, the aggregate function arguments are first inserted
/// into a sorter in the main loop, and in the group by aggregation phase
/// we read the data from the sorter.
///
/// In the alternative case, no sorting is required for group by,
/// and the aggregate function arguments are retrieved directly from
/// registers allocated in the main loop.
pub enum GroupByAggArgumentSource<'a> {
/// The aggregate function arguments are retrieved from a pseudo cursor
/// which reads from the GROUP BY sorter.
PseudoCursor {
cursor_id: usize,
col_start: usize,
dest_reg_start: usize,
aggregate: &'a Aggregate,
},
/// The aggregate function arguments are retrieved from a contiguous block of registers
/// allocated in the main loop for that given aggregate function.
Register {
src_reg_start: usize,
aggregate: &'a Aggregate,
},
}
```
Closes#1438
