Commit Graph

281 Commits

Author SHA1 Message Date
Pekka Enberg
bf7f80a937 core/io: Switch Unix I/O to use libc::pwrite()
We use libc elsewhere for fault injection reasons, so let's do this
call-site too.
2025-08-27 17:56:23 +03:00
PThorpe92
8c64b772e7 Use previous WindowsIO impl as generic IO 2025-08-25 19:04:14 -04:00
PThorpe92
177c717f25 Remove windows IO in place of Generic IO 2025-08-25 18:47:21 -04:00
Pekka Enberg
5fe5e1548b core/io: Fix build on Android and iOS
Commit ebe6aa0d28 ("adjust cfg for unix
and linux IO") adjusted the I/O conditional compilation, but forgot that
Android and iOS are also part of Unix target family.

Fixes #2500
2025-08-25 11:21:46 +03:00
rajajisai
9068a29380 Use unsafe block 2025-08-24 18:56:05 -04:00
rajajisai
84d20ba60f Use F_FULLSYNC in darwin based operating systems 2025-08-24 18:45:46 -04:00
Jussi Saurio
14873c76fb unixio: use Mutex::lock() instead of Mutex::try_lock()
we should wait to obtain the lock, not immediately fail if we cant.
2025-08-22 10:47:50 +03:00
Pekka Enberg
ae8b1eb00d Merge 'core/io: Don't open file as non-blocking in Unix backend' from Pekka Enberg
The Unix backend is a syscall()-based, blocking implementation. The
O_NONBLOCK adds nothing.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2708
2025-08-21 19:13:39 +03:00
Pekka Enberg
ce8b4c20f6 core/io: Don't open file as non-blocking in Unix backend
The Unix backend is a syscall()-based, blocking implementation. The
O_NONBLOCK adds nothing.
2025-08-21 14:43:00 +03:00
Nikita Sivukhin
f99843cc9e fix windows io 2025-08-21 14:57:07 +04:00
Nikita Sivukhin
c771487933 add remove_file method to the IO 2025-08-21 14:51:02 +04:00
Pekka Enberg
9233f48e08 core/io: Switch Unix I/O operations to use libc
We need it for LD_PRELOAD fault injection to work.
2025-08-20 13:43:47 +03:00
Pekka Enberg
c2208a542a Merge 'Initial pass to support per page encryption' from Avinash Sajjanshetty
This patch adds support for per page encryption. The code is of alpha
quality, was to test my hypothesis. All the encryption code is gated
behind a `encryption` flag. To play with it, you can do:
```sh
cargo run --features encryption -- database.db

turso> PRAGMA key='turso_test_encryption_key_123456';

turso> CREATE TABLE t(v);
```
Right now, most stuff is hard coded. We use AES GCM 256. This
information is not stored anywhere, but in future versions we will start
saving this info in the file. When writing to disk, we will generate a
cryptographically secure random salt, use that to encrypt the page. Then
we will store the authentication tag and the salt in the page itself. To
accommodate this encryption hardcodes reserved space of 28 bytes.
Once the key is set in the connection, we propagate that information to
pager and the WAL, to encrypt / decrypt when reading from disk.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2567
2025-08-20 11:11:24 +03:00
Avinash Sajjanshetty
bd9b4bbfd2 encrypt/decrypt when writing/reading from DB 2025-08-20 11:47:23 +05:30
pedrocarlo
f27d4d14f2 remove polling code in UnixIO so we can implement it correctly later and so we do not fool ourselves that we have any async code there that actually runs 2025-08-20 01:36:08 -03:00
Jussi Saurio
b5439dd068 Remove assertions from Completion::complete() and Completion::error()
The completion callback can be invoked only once via `OnceLock`, let's not
crash if we e.g. call `Completion::abort()` on an already finished completion.

Closes #2673
2025-08-19 22:02:02 +03:00
pedrocarlo
66171527b4 thread safely store the result of completion 2025-08-19 10:48:21 -03:00
pedrocarlo
de1811dea7 abort completions on error 2025-08-19 10:48:21 -03:00
pedrocarlo
4dca1c00db fix merge conflict 2025-08-19 10:48:21 -03:00
pedrocarlo
ab3b68e360 change completion callbacks to take a Result param + create separate functions to declare a completion errored 2025-08-19 10:48:21 -03:00
pedrocarlo
71ca221390 clippy 2025-08-19 10:48:21 -03:00
pedrocarlo
2d6fad5ea3 nit: adjust order of struct completions 2025-08-19 10:48:21 -03:00
pedrocarlo
fadf78fe67 use a dedicated Error enum for Completion Error 2025-08-19 10:48:21 -03:00
pedrocarlo
7bc0545442 default impl for get_memory_io 2025-08-19 10:48:21 -03:00
pedrocarlo
d5a59c6bee default impl for generate_random_number 2025-08-19 10:48:21 -03:00
pedrocarlo
f72bcbc5da default impl for wait_for_completion + check for errors in completion there 2025-08-19 10:48:21 -03:00
pedrocarlo
002390b5a5 store error inside Completion 2025-08-19 10:48:21 -03:00
pedrocarlo
d0c13f0104 remove IOError from Parser + store only ErrorKind in LimboError 2025-08-19 10:48:21 -03:00
PThorpe92
d3d01cefc8 Add to_system_time for our io::clock::Instant type 2025-08-18 19:27:37 -04:00
PThorpe92
cc2fed3297 Remove copy_to API from file IO trait 2025-08-14 21:31:13 -04:00
PThorpe92
55f09a01c4 Update copy_to method in file trait to separate source and destination IO 2025-08-14 21:31:13 -04:00
Preston Thorpe
5cea0f572e Merge 'Revive async io extension PR' from Preston Thorpe
bringing #1127 back to life, except better because this doesn't add
Tokio as a dependency for extension lib just for tests.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #2418
2025-08-14 16:10:09 -04:00
Preston Thorpe
e9d23e7fea apply copilot suggestion
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-08-14 16:08:55 -04:00
Jussi Saurio
a2a6feb193 Merge 'Use BufferPool owned by Database instead of a static global' from Jussi Saurio
## Problem
There are several problems with our current statically allocated
`BufferPool`.
1. You cannot open two databases in the same process with different page
sizes, because the `BufferPool`'s `Arena`s will be locked forever into
the page size of the first database. This is the case regardless of
whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.
2. It is impossible to even write Rust tests for different page sizes
because of this, assuming the test uses a single process.
## Solution
Make `Database` own `BufferPool` instead of it being statically
allocated, so this problem goes away.
Note that I didn't touch the still statically-allocated
`TEMP_BUFFER_CACHE`, because it should continue to work regardless of
this change. It should only be a problem if the user has two or more
databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a
time, so the rest of the allocations will go through the global
allocator instead.
## Notes
I extracted this change out from #2569, because I didn't want it to be
smuggled in without being reviewed as an individual piece.

Reviewed-by: Avinash Sajjanshetty (@avinassh)

Closes #2596
2025-08-14 12:40:32 +03:00
Jussi Saurio
78f1ed979e Merge 'io_uring: Gracefully handle submission queue overflow' from Preston Thorpe
Current handling is not ideal, this adds proper squeue overflow handling
by ensuring everything is still submitted in-order

Closes #2586
2025-08-14 10:55:17 +03:00
Jussi Saurio
359cba0474 Use BufferPool owned by Database instead of a static global
Problem

There are several problems with our current statically allocated
`BufferPool`.

1. You cannot open two databases in the same process with different
page sizes, because the `BufferPool`'s `Arena`s will be locked forever
into the page size of the first database. This is the case regardless
of whether the two `Database`s are open at the same time, or if the first
is closed before the second is opened.

2. It is impossible to even write Rust tests for different page sizes because
of this, assuming the test uses a single process.

Solution

Make `Database` own `BufferPool` instead of it being statically allocated, so this
problem goes away.

Note that I didn't touch the still statically-allocated `TEMP_BUFFER_CACHE`, because
it should continue to work regardless of this change. It should only be a problem if
the user has two or more databases with different page sizes open simultaneously, because
`TEMP_BUFFER_CACHE` will only support one pool of a given page size at a time, so the rest
of the allocations will go through the global allocator instead.

Notes

I extracted this change out from #2569, because I didn't want it to be smuggled in without
being reviewed as an individual piece.
2025-08-14 10:29:52 +03:00
PThorpe92
3840708970 Add safety comment 2025-08-13 21:42:19 -04:00
PThorpe92
0e3750b85b Integrate changes to vfs IO impl for callbacks 2025-08-13 21:42:18 -04:00
PThorpe92
ec4bf19fc7 Gracefully handle submission queue overflow in io_uring backend 2025-08-13 12:07:41 -04:00
Nikita Sivukhin
c3c7463193 add debug logs to the memory io 2025-08-13 15:23:18 +04:00
pedrocarlo
2e68296107 create IOCompletions 2025-08-13 10:24:55 +03:00
PThorpe92
213d589dd1 Apply review suggestions, remove FreeEntry 2025-08-08 11:07:29 -04:00
PThorpe92
faf248df03 Add more docs and comments for TempBufferCache 2025-08-08 10:55:28 -04:00
PThorpe92
34d90d5acb Remove Clone impl for Buffer and PageContent to make any copying of page data explicit 2025-08-08 10:55:28 -04:00
PThorpe92
d7e4ba21f8 Add explanation for using 3mb limit 2025-08-08 10:55:28 -04:00
PThorpe92
66964fd8d2 Add documentation and comments to new buffer pool, add get_frame api 2025-08-08 10:55:28 -04:00
PThorpe92
d94e252ef9 Actually pre-register two arenas with the ring 2025-08-08 10:55:28 -04:00
PThorpe92
39d230a899 Add bitmap for tracking pages in arena 2025-08-08 10:55:27 -04:00
PThorpe92
0ffba81216 Make register buffer io trait return the buf index 2025-08-08 10:55:27 -04:00
PThorpe92
fd09fe1237 Adjust io_uring to register two arenas, one for frames and the other for db pages 2025-08-08 10:55:27 -04:00