TLDR: no need to call either of:
---
program.emit_insn_with_label_dependency() -> just call
program.emit_insn()
program.defer_label_resolution() -> just call program.resolve_label()
---
Changes:
- make BranchOffset an explicit enum (Label, Offset, Placeholder)
- remove program.emit_insn_with_label_dependency() - label dependency is
automatically detected
- for label to offset mapping, use a hashmap from label(negative i32) to
offset (positive u32)
- resolve all labels in program.build()
- remove program.defer_label_resolution() - all labels are resolved in
build()
Closes#625
TLDR: no need to call either of:
program.emit_insn_with_label_dependency() -> just call program.emit_insn()
program.defer_label_resolution() -> just call program.resolve_label()
Changes:
- make BranchOffset an explicit enum (Label, Offset, Placeholder)
- remove program.emit_insn_with_label_dependency() - label dependency is automatically detected
- for label to offset mapping, use a hashmap from label(negative i32) to offset (positive u32)
- resolve all labels in program.build()
- remove program.defer_label_resolution() - all labels are resolved in build()
Lots of cleanup still left to do. Draft PR for adding support for OPFS
for WASM build (add support for limbo in browser).
Overall explanation of the architecture: this follows the sqlite wasm
architecture for OPFS.
main <> (limbo-worker.js) limbo (VFS - opfs.js) <> opfs-sync-proxy
The main thread loads limbo-worker.js which bootstraps the opfs.js and
opfs-sync-proxy.js and then launches the limbo-wasm.js.
At that point it can be used with worker.postmessage and
worker.onmessage interactions from the main thread.
The VFS provided by opfs.js provides a sync API by offloading async
operations to opfs-sync-proxy.js. This is done through SharedArrayBuffer
and Atomic.wait() to make the actual async operations appear synchronous
for limbo.
resolves#531Closes#594
I noticed that the parse errors were a bit hard to read - only the
nearest token and the line/col offsets were printed.
I made a first attempt at improving the errors using
[miette](https://github.com/zkat/miette).
- Added derive for `miette::Diagnostic` to both the parser's error type
and LimboError.
- Added miette dependency to both sqlite3_parser and core. The `fancy`
feature is only enabled for the CLI. So the overhead on the libraries
(core, parser) should be minimal.
Some future improvements that can be made further:
- Add spans to AST nodes so that errors can better point to the correct
token. See upstream issue: https://github.com/gwenn/lemon-rs/issues/33
- Construct more errors with offset information. I noticed that most
parser errors are constructed with `None` as the offset.
- The messages are a bit redundant (example "syntax error at (1, 6)").
This can improved.
Comparisons.
Before:
```
❯ cargo run --package limbo --bin limbo database.db --output-mode pretty
...
limbo> selet * from a;
[2025-01-05T11:22:55Z ERROR sqlite3Parser] near "Token([115, 101, 108, 101, 116])": syntax error
Parse error: near "selet": syntax error at (1, 6)
```
<img width="969" alt="image" src="https://github.com/user-
attachments/assets/82651a77-f5ac-4eee-b208-88c6ea7fc9b7" />
After:
```
❯ cargo run --package limbo --bin limbo database.db --output-mode pretty
...
limbo> selet * from a;
[2025-01-05T12:25:52Z ERROR sqlite3Parser] near "Token([115, 101, 108, 101, 116])": syntax error
× near "selet": syntax error at (1, 6)
╭────
1 │ selet * from a
· ▲
· ╰── syntax error
╰────
```
<img width="980" alt="image" src="https://github.com/user-
attachments/assets/747a90e5-5085-41f9-b0fe-25864179ca35" />
Closes#618
#514
Introduces a new feature for lazy evaluation in the
`Statement.raw().iterate()` method and includes related changes in both
the test and implementation files. The most important changes include
adding a test case for lazy evaluation, creating a `RowIterator` struct,
and modifying the `iterate` method to use this new struct.
Everything seems to works fine, but suggestions on code improvement and
test use cases are welcoming.
Closes#527
This PR adds the `datetime` function, with all the support currently
that date/time have for modifiers, and `julianday` function, as well as
some additional modifiers for date/time/datetime.
There are a couple considerations here, I left a couple comments but
essentially there is going to have to be some more work done to track
the state of the expression during the application of modifiers, to
handle a bunch of edge-cases like re-applying the same timezone modifier
to itself, or converting an integer automatically assumed to be
julianday, into epoch, or `ceiling`/`floor` which will determine
relative addition of time in cases like
```
2024-01-31 +1 month = 2024-03-02
```
which was painful enough to get working to begin with.
I couldn't get the `julianday_converter` library to get the exact same
float precision as sqlite, so function is included that matches their
output, for some reason floating point math + `.floor()` would give the
correct result. They seem to 'round' to 8 decimal places, and I was able
to get this to work with the same output as sqlite, except in cases like
`2234.5`, in which case we return `2234.5000000` because of the `fmt`
precision:
```rust
pub fn exec_julianday(time_value: &OwnedValue) -> Result<String> {
let dt = parse_naive_date_time(time_value);
match dt {
// if we did something heinous like: parse::<f64>().unwrap().to_string()
// that would solve the precision issue, but dear lord...
Some(dt) => Ok(format!("{:.1$}", to_julian_day_exact(&dt), 8)),
None => Ok(String::new()),
}
}
```
Suggestions would be appreciated on the float precision issue.
Reviewed-by: Sonny <14060682+sonhmai@users.noreply.github.com>
Closes#600
## Purpose of the PR
- Add minimal template code that provides Limbo features.
## Changes
- Added `DB` which is an interface to DB.
- Added 'LimboDB` which is a thin wrapper around native methods provided
using jni.
## TODO
- Incrementally update the code to support jdbc. Refer to [sqlite-
jdbc](https://github.com/xerial/sqlite-jdbc).
## Reference
- https://github.com/tursodatabase/limbo/issues/615Closes#619
- Removed unecessary clones
- Got rid of uneeded scoping in read_complete callback
- Use more rusty syntax e.g. std::cell::RefCell::borrow_mut(&buffer) ->
buffer.borrow_mut()
- Changed write_page.unwrap() to write_page()? to match run_once and
read_page()
No functional change.
Reviewed-by: Jussi Saurio <kacperoza@gmail.com>
Closes#608
Manually vendored in some changes from [lemon-
rs](https://github.com/gwenn/lemon-rs), including a merged change from
@krishvishal and [an unmerged PR ](https://github.com/gwenn/lemon-
rs/pull/81) from user ignatz that boxes Limit. I also boxed `OneSelect`
because it also improved perf in the benchmarks. 40-50% more throughput
with these changes to our existing admittedly simple benchmarks. Added a
new more complex prepare benchmark that includes group by and having as
well, which is also 42% faster with the new code.
**Runs on my local machine:**
```
main:
limbo/Prepare statement: 'SELECT 1'
time: [1.2902 µs 1.2927 µs 1.2958 µs]
thrpt: [771.73 Kelem/s 773.56 Kelem/s 775.07 Kelem/s]
change:
time: [+0.2770% +0.6013% +0.9243%] (p = 0.00 < 0.05)
thrpt: [-0.9158% -0.5977% -0.2762%]
limbo/Prepare statement: 'SELECT * FROM users LIMIT 1'
time: [2.4885 µs 2.4927 µs 2.4971 µs]
thrpt: [400.47 Kelem/s 401.18 Kelem/s 401.84 Kelem/s]
change:
time: [+1.2859% +1.6970% +2.0993%] (p = 0.00 < 0.05)
thrpt: [-2.0561% -1.6687% -1.2696%]
limbo/Prepare statement: 'SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1)...
time: [5.6867 µs 5.6994 µs 5.7164 µs]
thrpt: [174.93 Kelem/s 175.46 Kelem/s 175.85 Kelem/s]
change:
time: [+16.921% +17.332% +17.765%] (p = 0.00 < 0.05)
thrpt: [-15.085% -14.772% -14.472%]
this branch:
limbo/Prepare statement: 'SELECT 1'
time: [861.48 ns 862.60 ns 863.79 ns]
thrpt: [1.1577 Melem/s 1.1593 Melem/s 1.1608 Melem/s]
change:
time: [-33.293% -33.042% -32.754%] (p = 0.00 < 0.05)
thrpt: [+48.709% +49.347% +49.909%]
Performance has improved.
limbo/Prepare statement: 'SELECT * FROM users LIMIT 1'
time: [1.6080 µs 1.6106 µs 1.6140 µs]
thrpt: [619.58 Kelem/s 620.87 Kelem/s 621.88 Kelem/s]
change:
time: [-35.838% -35.611% -35.380%] (p = 0.00 < 0.05)
thrpt: [+54.750% +55.305% +55.857%]
Performance has improved.
Benchmarking limbo/Prepare statement: 'SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1)...: Collecting 100 samples in estimated 5.0125 s (1.
limbo/Prepare statement: 'SELECT first_name, count(1) FROM users GROUP BY first_name HAVING count(1)...
time: [4.0161 µs 4.0301 µs 4.0473 µs]
thrpt: [247.08 Kelem/s 248.13 Kelem/s 249.00 Kelem/s]
change:
time: [-29.791% -29.596% -29.399%] (p = 0.00 < 0.05)
thrpt: [+41.642% +42.038% +42.431%]
Performance has improved.
```
**Runs in CI:**
```
most recent commit on main:
limbo/Prepare statement: 'SELECT 1'
time: [2.7085 µs 2.7113 µs 2.7153 µs]
thrpt: [368.28 Kelem/s 368.83 Kelem/s 369.21 Kelem/s]
limbo/Prepare statement: 'SELECT * FROM users LIMIT 1'
time: [4.8688 µs 4.8713 µs 4.8741 µs]
thrpt: [205.17 Kelem/s 205.29 Kelem/s 205.39 Kelem/s]
this branch:
limbo/Prepare statement: 'SELECT 1'
time: [1.9278 µs 1.9329 µs 1.9405 µs]
thrpt: [515.33 Kelem/s 517.35 Kelem/s 518.73 Kelem/s]
limbo/Prepare statement: 'SELECT * FROM users LIMIT 1'
time: [3.5708 µs 3.5 µs 3.5794 µs]
thrpt: [279.38 Kelem/s 279.75 Kelem/s 280.05 Kelem/s]
```
**Discussion:**
Generally I think we should probably just, philosophically, hard fork
this vendored code and start making whatever modifications we want to
it... thoughts?
Also I guess there's a way to add a co-authored by XXX to these commits
so that they don't show up under my name only, because I didn't write
most of it.
Closes#620
I noticed that the parse errors were a bit hard to read - only the nearest token and the line/col offsets were printed.
I made a first attempt at improving the errors using [miette](https://github.com/zkat/miette).
- Added derive for `miette::Diagnostic` to both the parser's error type and LimboError.
- Added miette dependency to both sqlite3_parser and core. The `fancy` feature is only enabled for CLI.
Some future improvements that can be made further:
- Add spans to AST nodes so that errors can better point to the correct token. See upstream issue: https://github.com/gwenn/lemon-rs/issues/33
- Construct more errors with offset information. I noticed that most parser errors are constructed with `None` as the offset.
Comparisons.
Before:
```
❯ cargo run --package limbo --bin limbo database.db --output-mode pretty
...
limbo> selet * from a;
[2025-01-05T11:22:55Z ERROR sqlite3Parser] near "Token([115, 101, 108, 101, 116])": syntax error
Parse error: near "selet": syntax error at (1, 6)
```
After:
```
❯ cargo run --package limbo --bin limbo database.db --output-mode pretty
...
limbo> selet * from a;
[2025-01-05T12:25:52Z ERROR sqlite3Parser] near "Token([115, 101, 108, 101, 116])": syntax error
× near "selet": syntax error at (1, 6)
╭────
1 │ selet * from a
· ▲
· ╰── syntax error
╰────
```
Purpose of this PR
- Add support for Java (as Java has an extensive community)
- Enable Limbo to be provided as a Java library in the future.
Changes
- Added `bindings/java` directory.
- Created `src` package for Java (Gradle) and `rs_src` for Rust JNI
code.
- Implemented basic functionality to gather feedback and proceed with
further development.
- Some features just printout the result for testing purposes.
Future Work
- Integrate CI to publish the library.
- Enhance error handling mechanisms.
- Implement additional features and functionality.
- Add test code after we decide on which features to provide
Issue
https://github.com/tursodatabase/limbo/issues/615Closes#613
No functional changes, just move almost everything out of `emitter.rs`
into smaller modules with more distinct responsibilities. Also, from
`expr.rs`, move `translate_aggregation` into `aggregation.rs` and
`translate_aggregation_groupby` into `group_by.rs`
Closes#610
In sqlite3, before arithmetic operation is done, it first checks if the
operation dosent overflow and then does the operation. In case it
overflows it would covert the arguments into floats and then does the
operation as [per code](https://github.com/sqlite/sqlite/blob/ded37f337b
7b2e916657a83732aaec40eb146282/src/vdbe.c#L1875) . I have done the same
behaviour for limbo.
Closes#612
Closes#448
Adds support for:
- Automatically creating index on the PRIMARY KEY if the pk is not a
rowid alias
- Parsing the automatically created index into memory, for use in
queries
* `testing/testing_norowidalias.db` now uses the PK indexes and some
tests were failing -- looks like taking the index into use revealed some
bugs in our codegen :) I fixed those in later commits.
Does not add support for:
- Inserting to the index during writes to the table
Closes#588
We had not implemented arithmetic operations for text values. This PR
implements this and aligns the behavior with sqlite3 .
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#605
Changes:
---
Instead of passing around:
1. `SymbolTable` (for resolving functions dynamically provided by
extensions), and
2. `precomputed_exprs_to_registers` (for resolving registers
containing results from already-computed expressions),
add `Resolver` struct to `TranslateCtx` that encapsulates both.
This introduces some lifetime annotation spam unfortunately, but maybe
we can also migrate to using references instead of cloning elsewhere as
well, since we generally do a lot of copying of expressions right now
for convenience.
---
Use way less arguments to functions - mainly just passing `program`,
`t_ctx` and `plan` around.
Closes#609
