Commit Graph

5456 Commits

Author SHA1 Message Date
Pere Diaz Bou
d66c683a4c implement rollback translation 2025-06-25 13:45:32 +02:00
Pekka Enberg
325aa320ba Merge remote-tracking branch 'origin/main' 2025-06-25 14:41:26 +03:00
Pekka Enberg
15fed363c7 Update COMPAT.md 2025-06-25 14:06:46 +03:00
Pekka Enberg
90d9e5cc71 Merge 'stress: reopen db / reconnect to db every now and then' from Jussi Saurio
Closes #1821
Two caveats:
- I wouldn't merge this before #1761 since it fails constantly against
`main` code
- Not sure if this works as intended with multiple threads, but we don't
support that in `stress` yet anyway

Reviewed-by: Pekka Enberg <penberg@iki.fi>

Closes #1822
2025-06-25 12:57:07 +03:00
Pekka Enberg
b1f5dd8b3e Merge 'sim: add Fault::ReopenDatabase' from Jussi Saurio
Add fault type that reopens the DB and reconnects the connections. It
purposefully does not call conn.close(), as the default behavior of that
method is to checkpoint the database.
This easily exposes multiple manifestations of DB/WAL corruption caused
by this issue: https://github.com/tursodatabase/limbo/issues/1725
in fact, in my testing, every run fails when this fault is enabled.
Hence I've added the --disable-reopen-database flag if you want to try
to find some other bugs.

Closes #1762
2025-06-25 12:57:02 +03:00
Pekka Enberg
1221cab41e Merge 'Fix database header initialization' from Diego Reis
This PR makes limbo read the database header and schema via the regular
pager `read_page()` route, instead of always reading it from the
database file. This is very important for correctness as modifications
to page 1 (containing header and schema) may exist in WAL frames, not
just the main database file. Without this, practically any real use of
the database would very quickly corrupt the DB as limbo would read the
incorrect page count from the stale page 1 copy in the database file,
and e.g. allocate the same rootpage for multiple tables and so on.
This issue has been so far hidden due to lack of tests and the fact that
`Connection::close()` checkpoints the DB by default, which has delayed
discovering this incredibly basic issue since most manual testing has
happened via CLI, and if everything is always checkpointed, reopening
the DB doesn't cause any problems.
This closes #1725
See also: #1762 (commit from this PR is bundled into this branch as
well)
EDIT: Also closes #1818

Closes #1761
2025-06-25 12:56:56 +03:00
Jussi Saurio
480f0a04b5 make clippy happy about mutating database_size immediately after default construction 2025-06-24 14:41:50 -03:00
Jussi Saurio
f21cde9501 post-rebase fixes 2025-06-24 14:41:50 -03:00
Jussi Saurio
f306547ff3 booleans are hard 2025-06-24 14:41:50 -03:00
Jussi Saurio
920e88a6a9 clippy 2025-06-24 14:41:50 -03:00
Diego Reis
1921fcb943 Add comments to clarify current behaviour 2025-06-24 14:41:50 -03:00
Diego Reis
6ae196d7b3 Add mutex to allocating page1
This is to prevent race conditions where two threads could try to initialize database at the same time
2025-06-24 14:41:50 -03:00
Diego Reis
a1b7b3c6f6 Fix clippy complains 2025-06-24 14:41:50 -03:00
Jussi Saurio
a5d71a65be clippy doesnt get it 2025-06-24 14:41:50 -03:00
Jussi Saurio
133d498724 Implement a header_accessor module so that DatabaseHeader structs arent initialized on every access 2025-06-24 14:41:50 -03:00
Jussi Saurio
ad20e306f7 dont panic on cache clear error 2025-06-24 14:41:50 -03:00
Jussi Saurio
2dc9b1cf7f Fix: use either db_size>0 or wal max frame>0 to decide when to read schema 2025-06-24 14:41:50 -03:00
Jussi Saurio
ba3bfa058d Delete unused stuff 2025-06-24 14:41:50 -03:00
Jussi Saurio
6091d5abd7 Add todo comment 2025-06-24 14:41:50 -03:00
Jussi Saurio
fcc070c27e Remove unused method 2025-06-24 14:41:50 -03:00
Jussi Saurio
8d4ed110c0 alloc page1 on first tx (read OR write) - otherwise e.g. select * from sqlite_schema panics 2025-06-24 14:41:50 -03:00
Jussi Saurio
d05030ed64 fix autovacuum test 2025-06-24 14:41:50 -03:00
Jussi Saurio
4b5151d2c6 Fix not setting Database::maybe_shared_wal to Some after creating WAL 2025-06-24 14:41:50 -03:00
Diego Reis
43aa2577cd Implement size for JS and WASM storages 2025-06-24 14:41:50 -03:00
Diego Reis
af193ef76a Move first page allocation logic to begin_write_tx 2025-06-24 14:41:50 -03:00
Jussi Saurio
62e52e0148 only initialize pager in empty state when no WAL and db file size 0 2025-06-24 14:41:50 -03:00
Jussi Saurio
acafe71d9e Dont allocate fresh page1 if it's in the WAL 2025-06-24 14:41:50 -03:00
Jussi Saurio
dc13d88900 Read schema from nonempty WAL if present 2025-06-24 14:41:50 -03:00
Jussi Saurio
9aae1ff00a sim: add Fault::ReopenDatabase
Add fault type that reopens the DB and reconnects the connections.
It purposefully does not call conn.close(), as the default behavior
of that method is to checkpoint the database.

This easily exposes multiple manifestations of DB/WAL corruption caused by
this issue: https://github.com/tursodatabase/limbo/issues/1725

in fact, in my testing, every run fails when this fault is enabled. Hence I've
added the --disable-reopen-database flag if you want to try to find some other
bugs.
2025-06-24 14:41:50 -03:00
Diego Reis
f0f9ad4844 core: Get rid of maybe_init_database_file
Initialization now only occurs in the first write transaction
2025-06-24 14:41:50 -03:00
Diego Reis
ce73025202 core: Only actually allocate page1 on first write transaction 2025-06-24 14:41:50 -03:00
Diego Reis
2f33c799e3 core: Set default database_size to 0 2025-06-24 14:41:49 -03:00
Diego Reis
75fdbd73c6 core: Add count of pages in Pager and fix page1 initialization 2025-06-24 14:41:49 -03:00
Diego Reis
9c7330c01c core: Add size method to DatabaseStorage trait 2025-06-24 14:41:49 -03:00
Jussi Saurio
cc2e14b11c Read page 1 from pager always, no separate db_header 2025-06-24 14:41:49 -03:00
Jussi Saurio
4999d0ed95 stress: reopen db / reconnect to db every now and then 2025-06-24 16:34:13 +03:00
Pekka Enberg
f01516378b Merge 'Add Pedro to email recipients for antithesis' from Pedro Muniz
I want to be notified on Antithesis test stuff. I want to understand
more how their stuff works and what fails and how in their platform.

Closes #1785
2025-06-24 16:12:31 +03:00
Pekka Enberg
2313c33015 Merge 'bindings/rust: Implement Debug for Connection' from Charlie
A simple change to implement the `Debug` trait for the `Connection`,
similar to how it is implemented for `Database`.  This should help users
in their application code with wrapping the connection.

Closes #1798
2025-06-24 16:11:54 +03:00
Pekka Enberg
b1cd7229c7 Merge 'sim: when loading bug, dont panic if there are no runs' from Jussi Saurio
Closes #1820
2025-06-24 16:07:52 +03:00
Pekka Enberg
69d8162e61 Merge 'sim: fix singlequote escaping and unescaping' from Jussi Saurio
Fixes https://github.com/tursodatabase/limbo/issues/1801 by preventing
the generation of strings like `'foo''` that contain an improper
unescaped sequence of quotes

Closes #1819
2025-06-24 16:07:43 +03:00
Pekka Enberg
28c2d24931 antithesis: Relax connect failure check
We also want to just ignore this under Antithesis:

``´
limbo.DatabaseError: Failed to open database: LockingError("Failed locking file. File is locked by another process")
```
2025-06-24 15:47:21 +03:00
Jussi Saurio
73db450b8b sim: when loading bug, dont panic if there are no runs 2025-06-24 15:46:15 +03:00
Jussi Saurio
cff96964df Fix singlequote escaping and unescaping 2025-06-24 14:58:25 +03:00
Pekka Enberg
6c8aa4e043 antithesis: Run experiment for 60 minutes 2025-06-24 14:16:09 +03:00
Pekka Enberg
f8c2c7387f antithesis: Reduce number of stress iterations
...currently takes a very long time to run.
2025-06-24 14:15:46 +03:00
Jussi Saurio
0a94ee5572 Merge 'Fix btree balance and seek after overwritten cell overflows' from Jussi Saurio
Closes #1809
Closes #1810
Closes #1811
Closes #1812
Closes #1813
Closes #1814
Closes #1815
and probably Closes #1805

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1816
2025-06-24 13:29:53 +03:00
Jussi Saurio
27b2c2530b test/stress: only do integrity_check every 100 queries 2025-06-24 13:13:31 +03:00
Jussi Saurio
bdfbb8fe54 Fix erroneous early return 2025-06-24 11:26:00 +03:00
Jussi Saurio
5878724d0e fix/btree: balance and seek after overwritten cell overflows 2025-06-24 11:08:22 +03:00
Pekka Enberg
a99833ba8d Merge 'Bring back some merge conflicts code' from Pedro Muniz
Due to some merge conflicts, some recent code from stress testing was
lost. Bringing it back here

Closes #1808
2025-06-24 09:07:32 +03:00