Commit Graph

5293 Commits

Author SHA1 Message Date
meteorgan
caa1801ec4 cargo fmt 2025-06-16 22:09:43 +08:00
meteorgan
4f742a3a0f Implement RowData opcode 2025-06-16 22:07:35 +08:00
Pekka Enberg
15daa9e81a Merge 'stress: Run integrity check for every iteration' from Pekka Enberg
Depends on https://github.com/tursodatabase/limbo/pull/1754

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1755
2025-06-16 16:31:00 +03:00
Pekka Enberg
db4945eada Merge 'Fix update queries to set n_changes ' from Kim Seon Woo
- `Update` query doesn't update `n_changes`. Let's make it work
- Add `InsertFlags` to add meta information related to insert operations
- For update query, add `UPDATE` flag
- Currently, the update query executes `Insn::Delete` and `Insn::Insert`
internally, it increases `n_change` by 2. So, for the update query,
let's skip increasing `n_change` for the `Insn::Insert`
https://github.com/tursodatabase/limbo/issues/1681

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1683
2025-06-16 16:30:20 +03:00
Pekka Enberg
d148f66670 stress: Make sure we run integrity check after every step
Spotted by Jussi.
2025-06-16 16:18:52 +03:00
Pekka Enberg
08bb4665b6 stress: Run integrity check for every iteration 2025-06-16 15:08:49 +03:00
Pekka Enberg
23b2d59871 Merge 'bindings/rust: Fix Rows::next() I/O dispatcher handling' from Pekka Enberg
The `next()` function needs to be a loop to make sure we actually return
rows.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1754
2025-06-16 15:08:34 +03:00
Pekka Enberg
f594a9db5a Merge 'core: Clean up integrity_check()' from Pekka Enberg
Suggested by Jussi.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1753
2025-06-16 15:08:19 +03:00
Pekka Enberg
4496a0d08a core: Clean up integrity_check()
Suggested by Jussi.
2025-06-16 14:46:36 +03:00
Pekka Enberg
47e08d34bf bindings/rust: Fix Rows::next() I/O dispatcher handling
The `next()` function needs to be a loop to make sure we actually return rows.
2025-06-16 14:28:08 +03:00
Pekka Enberg
882c5ca168 Merge 'Simple integrity check on btree' from Pere Diaz Bou
This PR adds support for the instruction `IntegrityCk` which performs an
integrity check on the contents of a single table. Next PR I will try to
implement the rest of the integrity check where we would check indexes
containt correct amount of data and some more.
<img width="1151" alt="image" src="https://github.com/user-
attachments/assets/29d54148-55ba-480f-b972-e38587f0a483" />

Closes #1719
2025-06-16 13:46:26 +03:00
Pekka Enberg
8c100f03fb Merge 'Make SQLite bindings thread-safe' from Pekka Enberg
Wrap the `sqlite3` object members with a mutex like SQLite does to make
it thread-safe. Note that as future work we should make the threading
model configurable as per SQLite.

Closes #1748
2025-06-16 11:48:00 +03:00
Pekka Enberg
96c47667ea Merge 'Add abbreviated alias for .quit and .exit' from Krishna Vishal
Accidentally found that you can quit SQLite CLI just by typing `.q`,
`.qu` and `.qui` instead of full `.quit`.
IMO this will be an improvement to the DX.
I've also found bunch of other dot commands that work with abbreviated
aliases in SQLite. If this PR is okay then I will also add them here.

Closes #1720
2025-06-16 11:11:17 +03:00
Pekka Enberg
5c35810f78 sqlite3: Make Clippy happy 2025-06-16 11:07:39 +03:00
Pekka Enberg
4d6b5c2b88 Merge 'cli: fix panic of queries with less than 7 chars' from Nils Koch
Hey everyone! I think this project is really really cool and I am here
to answer @jussisaurio
[call](https://x.com/jussisaur/status/1930290889496129562).
I started tinkering around with limbo and ran into this panic. Any input
with less than 7 characters (`"explain".len()`) will trigger it. In my
case it was `show;`. The issue was introduced a week ago in #1678.
PS: As I [said on X](https://x.com/nilskch/status/1930362930605723818),
I would love to help you with Limbo. I have a lot of experience with
DSLs and LSPs, and I have recently become interested in databases. I
don't have much experience with databases yet, but I am happy to learn.
I see that you have a lot of stuff going on with DSLs at the moment, so
I would be productive straight away in that area. Happy to chat if you
need help :)

Closes #1730
2025-06-16 11:06:02 +03:00
Pekka Enberg
dd001f9444 Merge 'Return parse error instead of corrupt error for no such table' from Pedro Muniz
This is more in line with SQLite:
```sql
sqlite> insert into t values (randomblob(1024*1024 * 6));
Parse error: no such table: t
```

Reviewed-by: Preston Thorpe (@PThorpe92)

Closes #1744
2025-06-16 11:04:50 +03:00
Pekka Enberg
5b47c9932d sqlite3: Make sqlite3 thread safe with a mutex
Make the sqlite3 object thread-safe with a mutex, like SQLite does.
2025-06-16 11:03:21 +03:00
Pekka Enberg
5fb50245ed sqlite3: Clean up sqlite3_wal_checkpoint() function 2025-06-16 11:02:44 +03:00
Pekka Enberg
50fdf7105d sqlite3: Clean up sqlite3_extended_errcode() function 2025-06-16 11:02:44 +03:00
Pekka Enberg
6901b7bcd6 sqlite3: Clean up sqlite3_errmsg() function 2025-06-16 11:02:44 +03:00
Pekka Enberg
0830578ec5 sqlite3: Clean up sqlite3_errcode() function 2025-06-16 11:02:44 +03:00
Pekka Enberg
5477d0a4f5 sqlite3: Clean up sqlite3_safety_check_sick_or_ok() function
Underscore in parameter or variable name indicates it's not used. Also,
drop useless `crate` qualifier from constants.
2025-06-16 11:02:44 +03:00
Pekka Enberg
0b0f712575 sqlite3: Move sqlite3_safety_check_sick_or_ok() to lib.rs
It's pointless to have a module for just one function...
2025-06-16 11:02:43 +03:00
Pekka Enberg
2b51fd69e8 Merge 'Switch Connection to use Arc instead of Rc' from Pekka Enberg
Connection needs to be Arc so that bindings can wrap it with `Mutex` for
multi-threading.

Closes #1749
2025-06-16 11:02:32 +03:00
Pekka Enberg
90c1e3fc06 Switch Connection to use Arc instead of Rc
Connection needs to be Arc so that bindings can wrap it with `Mutex` for
multi-threading.
2025-06-16 10:43:19 +03:00
PThorpe92
7e9a25ebca Merge 'simulator: disable all ansi encodings for debug print log file' from Pedro Muniz
This solves the issue of having weird ansi escape codes when creating
github issues for the simulator

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe (@PThorpe92)

Closes #1745
2025-06-15 19:00:18 -04:00
PThorpe92
28b8e1bce4 Merge 'Drop unused code in op_column' from meteorgan
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Preston Thorpe (@PThorpe92)

Closes #1743
2025-06-15 14:53:58 -04:00
pedrocarlo
6dfd4700e9 simulator: disable all ansi encodings for debug print log file 2025-06-14 19:34:40 -03:00
pedrocarlo
fef9add773 emit parse error instead of corrupt error for no such table 2025-06-14 18:45:23 -03:00
meteorgan
b910fd0012 drop unused code in op_column 2025-06-15 00:50:16 +08:00
Nils Koch
733c3603f2 cli: fix panic of queries with less than 7 chars 2025-06-13 20:25:47 +01:00
Jussi Saurio
30e4511d62 Merge 'NOT NULL constraint' from Anton Harniakou
Implements basic support for `NOT NULL` contraint check.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1675
2025-06-13 14:25:24 +03:00
Anton Harniakou
25d0584b19 Fix typo 2025-06-13 11:57:39 +03:00
Jussi Saurio
12563f878c Merge 'Change integrity check failure to say 'simulation failed:'' from Jussi Saurio
our AWS nightly sim looks for a line 'simulation failed:' in the process
output to capture context for the github issues it creates.
if the sim passes otherwise but integrity check fails, then this log
line must also be present there, otherwise the aws sim will panic due to
not finding a suitable line.

Closes #1716
2025-06-13 11:26:45 +03:00
Jussi Saurio
f7fa56ecfd Change integrity check failure to say 'simulation failed:'
our AWS nightly sim looks for a line 'simulation failed:' in the process
output to capture context for the github issues it creates.

if the sim passes otherwise but integrity check fails, then this log line
must also be present there, otherwise the aws sim will panic due to not finding
a suitable line.
2025-06-13 11:26:04 +03:00
Jussi Saurio
3f90fad131 Merge 'Simulator Ast Generation + Simulator Unary Operator + Refactor to use limbo_core::Value in Simulator for massive code reuse' from Pedro Muniz
This PR is a Drop-In replacement to the Predicate defined in the
Simulator. Predicate is basically the same as our ast::Expr, but it
supports a small number of the SQL expression syntax. By creating a
NewType that wraps ast::Expr we can tap into our already mostly
correctly defined parser structs. This change will enable us to easily
add generation for more types of sql queries.
I also added an ArbitraryFrom impl for ast::Expr that can be used in a
freestyle way (for now) for differential testing.
This PR also aims to implement Unary Operator logic similar to the
Binary Operator logic we have for predicate. After this change we may
need to adjust the Logic for how some assertions are triggered.
<s>Sometimes the `Select-Select-Optimizer` property thinks that these
two queries should return the same thing:
```sql
SELECT (twinkling_winstanley.sensible_federations > x'66616e7461737469625e0f37879823db' AND twinkling_winstanley.sincere_niemeyer < -7428368947470022783) FROM twinkling_winstanley WHERE 1;

SELECT * FROM twinkling_winstanley WHERE twinkling_winstanley.sensible_federations > x'66616e7461737469625e0f37879823db' AND twinkling_winstanley.sincere_niemeyer < -7428368947470022783;
```
However after running the shrunk plan manually, the simulator was
incorrect in asserting that. Maybe this a bug a in the generation of
such a query? Not sure yet. </s>
<b>EDIT: The simulator was correctly catching a bug and I thought I was
the problem. The bug was in `exec_if` and I fixed it in this PR.</b>
I still need to expand the Unary Operator generation to other types of
predicates. For now, I just implemented it for `SimplePredicate` as I'm
trying to avoid to bloat even more this PR.
<b>EDIT: I decided to just have one PR open for all the changes I'm
making to make my life a bit easier and to avoid merge conflicts with my
own branches that I keep spawning for new code.</b>
PS: This should only be considered for merging after
https://github.com/tursodatabase/limbo/pull/1619 is merged. Then, I will
remove the draft status from this PR.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1674
2025-06-13 11:23:10 +03:00
Jussi Saurio
0e5abb41c6 Merge 'Add manual WAL sync before checkpoint in con.close, fix async bug in checkpoint' from Preston Thorpe
In the normal path, we properly fsync the wal file before beginning a
checkpoint. However when we forcefully checkpoint upon closing the
connection we miss this step.

Closes #1695
2025-06-13 11:15:39 +03:00
PThorpe92
33b52bfb84 Replace refcel in wal sync, add counter timeout to conn close shutdown 2025-06-13 11:12:42 +03:00
PThorpe92
e134bd19da Remove close from Drop impl 2025-06-13 11:11:30 +03:00
PThorpe92
eecf6ae6e6 Wait till we write the page to increment current page in wal checkpoint 2025-06-13 11:11:30 +03:00
PThorpe92
9f966910bc Add manual wal sync before checkpoint in connection Drop 2025-06-13 11:11:30 +03:00
Jussi Saurio
0cfcf0f5cb Merge 'Refactor compound select' from meteorgan
Implement recursive processing for compound SELECT statements by
dividing them into two components:
1. left: a vector containing a SELECT core and its associated compound
operator
2. right_most: the rightmost SELECT core in the compound statement
Based on  the compound operator, we can determine how to process the
rightmost part and recursively handle the left component.
Following this refactor, we can seamlessly integrate support for
`INTERSECT` and `EXCEPT` operations.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1718
2025-06-13 11:08:31 +03:00
meteorgan
c0d260fd54 revise some comments 2025-06-13 10:39:36 +03:00
meteorgan
fd09675d8c clean up 2025-06-13 10:39:36 +03:00
meteorgan
6179d8de23 refactor compound select 2025-06-13 10:39:32 +03:00
Jussi Saurio
efed4bc07d Merge 'Complete ALTER TABLE implementation' from Levy A.
Resolves #895
- [x] `ALTER TABLE _ ADD _`
- [x] `ALTER TABLE _ DROP _`
- [x] `ALTER TABLE _ RENAME _ TO _`

Reviewed-by: Preston Thorpe (@PThorpe92)
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1527
2025-06-13 10:23:43 +03:00
Jussi Saurio
3c1765cbde Merge 'Fix large inserts to unique indexes hanging' from Jussi Saurio
Closes #1717
We were incorrectly setting `moved_before` as `false` after checking for
unique constraint violation, but the reality is that after the
uniqueness check, we are already correctly positioned -- if no match was
found, the cursor is positioned at either:
a) no row, or
b) at the first entry that is greater than the key we are inserting.
This means we don't have to move anymore and can just insert.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1723
2025-06-12 13:12:56 +03:00
Jussi Saurio
04c590f5a6 Add comment about re-entrancy of op_idx_insert() 2025-06-12 12:36:53 +03:00
Jussi Saurio
bdbd021bbb Fix large inserts to unique indexes hanging
We were incorrectly setting `moved_before` as `false` after checking
for unique constraint violation, but the reality is that after the
uniqueness check, we are already correctly positioned -- if no match
was found, the cursor is positioned at either:

a) no row, or
b) at the first entry that is greater than the key we are inserting.

This means we don't have to move anymore and can just insert.
2025-06-12 12:25:17 +03:00
Jussi Saurio
b3b32417e7 Merge 'betters instrumentation for btree related operations + cleaner debug for RefValue' from Pedro Muniz
Adds an `#[instrument]` to many btree related functions. Also, when
debugging #1717 it would be impossible to debug as the 10Mb SeekKey
would be printed to my terminal making the experience miserable. The
debug implementation now truncates the Blobs and Strings when they are
too big to be useful.

Closes #1721
2025-06-12 10:39:13 +03:00