Commit Graph

5426 Commits

Author SHA1 Message Date
Jussi Saurio
9aae1ff00a sim: add Fault::ReopenDatabase
Add fault type that reopens the DB and reconnects the connections.
It purposefully does not call conn.close(), as the default behavior
of that method is to checkpoint the database.

This easily exposes multiple manifestations of DB/WAL corruption caused by
this issue: https://github.com/tursodatabase/limbo/issues/1725

in fact, in my testing, every run fails when this fault is enabled. Hence I've
added the --disable-reopen-database flag if you want to try to find some other
bugs.
2025-06-24 14:41:50 -03:00
Diego Reis
f0f9ad4844 core: Get rid of maybe_init_database_file
Initialization now only occurs in the first write transaction
2025-06-24 14:41:50 -03:00
Diego Reis
ce73025202 core: Only actually allocate page1 on first write transaction 2025-06-24 14:41:50 -03:00
Diego Reis
2f33c799e3 core: Set default database_size to 0 2025-06-24 14:41:49 -03:00
Diego Reis
75fdbd73c6 core: Add count of pages in Pager and fix page1 initialization 2025-06-24 14:41:49 -03:00
Diego Reis
9c7330c01c core: Add size method to DatabaseStorage trait 2025-06-24 14:41:49 -03:00
Jussi Saurio
cc2e14b11c Read page 1 from pager always, no separate db_header 2025-06-24 14:41:49 -03:00
Pekka Enberg
f01516378b Merge 'Add Pedro to email recipients for antithesis' from Pedro Muniz
I want to be notified on Antithesis test stuff. I want to understand
more how their stuff works and what fails and how in their platform.

Closes #1785
2025-06-24 16:12:31 +03:00
Pekka Enberg
2313c33015 Merge 'bindings/rust: Implement Debug for Connection' from Charlie
A simple change to implement the `Debug` trait for the `Connection`,
similar to how it is implemented for `Database`.  This should help users
in their application code with wrapping the connection.

Closes #1798
2025-06-24 16:11:54 +03:00
Pekka Enberg
b1cd7229c7 Merge 'sim: when loading bug, dont panic if there are no runs' from Jussi Saurio
Closes #1820
2025-06-24 16:07:52 +03:00
Pekka Enberg
69d8162e61 Merge 'sim: fix singlequote escaping and unescaping' from Jussi Saurio
Fixes https://github.com/tursodatabase/limbo/issues/1801 by preventing
the generation of strings like `'foo''` that contain an improper
unescaped sequence of quotes

Closes #1819
2025-06-24 16:07:43 +03:00
Pekka Enberg
28c2d24931 antithesis: Relax connect failure check
We also want to just ignore this under Antithesis:

``´
limbo.DatabaseError: Failed to open database: LockingError("Failed locking file. File is locked by another process")
```
2025-06-24 15:47:21 +03:00
Jussi Saurio
73db450b8b sim: when loading bug, dont panic if there are no runs 2025-06-24 15:46:15 +03:00
Jussi Saurio
cff96964df Fix singlequote escaping and unescaping 2025-06-24 14:58:25 +03:00
Pekka Enberg
6c8aa4e043 antithesis: Run experiment for 60 minutes 2025-06-24 14:16:09 +03:00
Pekka Enberg
f8c2c7387f antithesis: Reduce number of stress iterations
...currently takes a very long time to run.
2025-06-24 14:15:46 +03:00
Jussi Saurio
0a94ee5572 Merge 'Fix btree balance and seek after overwritten cell overflows' from Jussi Saurio
Closes #1809
Closes #1810
Closes #1811
Closes #1812
Closes #1813
Closes #1814
Closes #1815
and probably Closes #1805

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1816
2025-06-24 13:29:53 +03:00
Jussi Saurio
27b2c2530b test/stress: only do integrity_check every 100 queries 2025-06-24 13:13:31 +03:00
Jussi Saurio
bdfbb8fe54 Fix erroneous early return 2025-06-24 11:26:00 +03:00
Jussi Saurio
5878724d0e fix/btree: balance and seek after overwritten cell overflows 2025-06-24 11:08:22 +03:00
Pekka Enberg
a99833ba8d Merge 'Bring back some merge conflicts code' from Pedro Muniz
Due to some merge conflicts, some recent code from stress testing was
lost. Bringing it back here

Closes #1808
2025-06-24 09:07:32 +03:00
pedrocarlo
13a55948d5 bring back some merge conflicts code 2025-06-23 23:01:28 -03:00
Jussi Saurio
e2d8d11738 Merge 'chore: fix clippy warnings' from Nils Koch
There are a many clippy warnings in the main branch. This PR addresses
some of them.
I will fix the rest of the warnings in follow up PRs. Furthermore, the
clippy CI check is not working. I can look into that after I fixed the
rest of the clippy warnings. After that, I think it would make sense to
migrate to the latest Rust version. Does that work for you?
EDIT: This PR resolves all Clippy warnings and enables the CI to run all
Clippy checks. The CI has extra checks for the `limbo-wasm` module. This
module still has some Clippy warnings that need attention.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1742
2025-06-23 23:22:21 +03:00
Nils Koch
4c7fcafe77 ci: fail on all clippy warnings 2025-06-23 19:52:13 +01:00
Nils Koch
2827b86917 chore: fix clippy warnings 2025-06-23 19:52:13 +01:00
Pekka Enberg
1a0d4f500f Merge 'simulator: integrity check per query' from Pedro Muniz
Now that we have our own integrity check, we can also execute it per
query to catch bugs earlier. We could use `Sqlite`s integrity check
instead of the one @pereman2 implemented, but I think this could also
help us battle test the integrity check he wrote.
Motivation for this PR came from this issue #1781 . I wanted to find a
minimal repo for that problem, so I introduced this check per query. The
nice thing is that we get some shrinking as well by using the simulator
instead of `limbo_stress`.
The following command will allow you to find the failing seed I posted
in the issue:
`cargo run -p limbo_sim -- --minimum-tests 10 --maximum-tests 100
--disable-create-index --seed 17321136847878547364`

Closes #1802
2025-06-23 20:17:32 +03:00
Pekka Enberg
8686d9bb49 Merge 'don't emit Delete for UPDATE statement' from Pere Diaz Bou
Previously we implemented update as a simple `Delete` + `Insert`
procedure which seemed okay for the moment but it wasn't. `Delete` can
trigger balance and a post balance `seek` which will leave cursor
pointing to an invalid page which `Insert` will try to insert to.
We solve this by removing `Delete` from the execution plan and rely on
`Insert` to properly overwrite the cell where the rowid is the same as
the one we are inserting.

Closes #1803
2025-06-23 20:16:16 +03:00
Pere Diaz Bou
404b5fad8a clippy 2025-06-23 18:19:44 +02:00
Pere Diaz Bou
b7fd4f1e18 update nchanges on op_insert 2025-06-23 18:11:58 +02:00
Pekka Enberg
43d8799f55 Merge 'stress: Improve progress reporting' from Pekka Enberg
A stress run can take a long time. Improve progress reporting to improve
DX.

Closes #1796
2025-06-23 17:48:55 +03:00
Pekka Enberg
c0fd559f92 Merge 'Fix: add uv sync to all packages for pytest github action' from Pedro Muniz
Hopefully, this solves the problem in main branch.

Closes #1800
2025-06-23 17:48:44 +03:00
Pekka Enberg
963d7b7daf stress: Improve progress reporting
A stress run can take a long time. Improve progress reporting to improve
DX.
2025-06-23 17:25:38 +03:00
Pere Diaz Bou
c7e9b3a546 don't emit Delete for UPDATE statement
Previously we implemented update as a simple `Delete` + `Insert`
procedure which seemed okay for the moment but it wasn't. `Delete` can
trigger balance and a post balance `seek` which will leave cursor
pointing to an invalid page which `Insert` will try to insert to.

We solve this by removing `Delete` from the execution plan and rely on
`Insert` to properly overwrite the cell where the rowid is the same as
the one we are inserting.
2025-06-23 15:44:34 +02:00
pedrocarlo
c9b11dd173 simulator integrity check per query 2025-06-22 21:47:17 -03:00
pedrocarlo
efc1516d7b add uv sync to all packages 2025-06-22 15:45:07 -03:00
CM-IV
3bb0374401 impl Debug for Connection 2025-06-22 11:41:07 -05:00
Pekka Enberg
b07e95b892 Merge 'Fix CI errors' from Piotr Rżysko
Fix CI errors caused by https://github.com/tursodatabase/limbo/pull/1741

Closes #1794
2025-06-22 09:24:12 +03:00
Piotr Rzysko
ddd5417978 Add sqlite_test_ext to Dockerfile.antithesis 2025-06-21 20:07:26 +02:00
Piotr Rzysko
0f3efe2fbb Copy sqlite_test_ext to Dockerfile.simulator 2025-06-21 19:51:11 +02:00
Piotr Rzysko
dda1ee86e2 Fix ruff errors 2025-06-21 19:32:16 +02:00
Jussi Saurio
d0f9df1f97 Merge 'Improve extension compatibility testing' from Piotr Rżysko
Extracted from https://github.com/tursodatabase/limbo/pull/1727.

Reviewed-by: Preston Thorpe (@PThorpe92)

Closes #1741
2025-06-21 19:09:33 +03:00
Jussi Saurio
a549f2971d Merge 'Ephemeral Table in Update' from Pedro Muniz
Closes #1713. Adds ephemeral table when a rowid_alias is being updated.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1726
2025-06-21 19:07:32 +03:00
Jussi Saurio
cac23d5b31 Merge 'Fix infinite aggregation loop when sorting is not required' from Piotr Rżysko
Previously, with the `index_experimental` feature enabled, the query in
the added test would enter an infinite loop. This happened because
`label_grouping_agg_step` pointed to a constant argument that was moved
to the end of the program. As a result, the aggregation loop would jump
to the constant, then return to the start of the main loop, rewind the
index, and re-enter the aggregation loop - causing it to repeat
indefinitely.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1792
2025-06-21 19:06:44 +03:00
Piotr Rzysko
64b83a45e8 Fix infinite aggregation loop when sorting is not required
Previously, with the `index_experimental` feature enabled, the query in
the added test would enter an infinite loop. This happened because
`label_grouping_agg_step` pointed to a constant argument that was moved
to the end of the program. As a result, the aggregation loop would jump
to the constant, then return to the start of the main loop, rewind the
index, and re-enter the aggregation loop—causing it to repeat
indefinitely.
2025-06-21 10:03:10 +02:00
pedrocarlo
9ae4f6ec40 fix merge conflict problems 2025-06-20 16:38:10 -03:00
pedrocarlo
6596ee28a8 introduce EphemeralTable query destination 2025-06-20 16:30:21 -03:00
pedrocarlo
e53a290a48 move ephemeral table logic to update plan and reuse select logic for ephemeral index 2025-06-20 16:30:21 -03:00
pedrocarlo
b3351dc709 tests + adjustment to halt error message 2025-06-20 16:29:10 -03:00
pedrocarlo
9048ad398b modify loop functions to accomodate for ephemeral tables 2025-06-20 16:29:10 -03:00
pedrocarlo
eda9d20a0b tests 2025-06-20 16:28:10 -03:00