This adds support for running the simulator under Miri to detect UB.
There are a few things to note about Miri and its limitations
- It has limited `libc` coverage, so it's not really possible to have
Miri help with `UringIO`/`UringFile` or `UnixIO`/`UnixFile`. That's a
big gap ☹️
- It **can** work for `GenericIO`/`GenericFile`, which only uses `std`
- It can't call external C libraries, so even using `sqlite` is out
(hence adding `--disable-integrity-check` to the simulator for Miri use)
- It runs on nightly, consequently there are a few new lints that don't
exist on turso's pinned version of rustc
Some questions I have about this MR
- I made `GenericFile::{lock_file,unlock_file}` noops so I could use
`GenericIO`. This isn't great, but if/when you update from Rust 1.88.0
to 1.89.0, `std::File::{lock,lock_shared,unlock}` will be stabilized and
available. Should I note that as a TODO or something?
- Previously, the sim runner shelled out to `git` to get stuff like the
current git hash and the repo directory. For Miri, that's out, and so is
`git2`. Unfortunately, `gix` is also out since it has a required
dependency that uses inline assembly, which Miri doesn't like. I wrote a
hacky shim that uses only std to look for `.git` and find the hash that
HEAD is pointing to. It doesn't deal with stuff like packed-refs or the
repo being a secondary one made with `git worktree`. I'm happy to
support that, but wanted to hear from maintainers before doing more
work.
Two UB occurrences I already found:
- `TursoRwLock::read` used `AtomicU64::compare_exchange_weak`, which is
(evidently) [allowed to spuriously fail](https://doc.rust-lang.org/std/s
ync/atomic/struct.AtomicU64.html#method.compare_exchange_weak) in
exchange for perf. Miri forces this behavior, which triggers trivial
read deadlocks even with zero readers/writers. I changed it to
`compare_exchange`, but I'm not an atomics expert.
- Uninitialized read in non-Unix
`core::storage::buffer_pool::arena::alloc`. This is a simple one,
resolved by using `std::alloc::alloc_zeroed` instead of
`std::alloc::alloc`
Moving forward, I'd be interested in potentially getting the tests to
run in Miri, too. `tokio` looks like a good example of a project with
partial coverage that runs it where they can. They have some extra test
config to allow as many as possible to run under Miri, with
appropriately scaled-down parameter values since Miri is super slow
Closes#3720
We currently return the exact same error from two different IdxDelete
paths. Improve the messages with context about what we're doing to make
this error more debuggable.
I've added a trait called `CursorTrait`. I know it's not a good name for
now, but I didn't know what tto change then enum `Cursor` to. This trait
wraps all common functionality, and some functionality that is yet too
specific that needs to be fixed.
This is needed in order to have layered cursors where for example,
MvccCursor will need a fallback BTreeCursor.
Closes#3660
Prevents something like `WHERE x = 5 AND x = 5` from becoming a two
component index key.
Closes#3656
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#3658
This PR introduces sparse vectors support and jaccard distance
implementation.
Also, this PR restructure the code to have all vector operations in
separate files (they grow pretty quickly as new vector representations
added to the DB).
Closes#3647
Various little fixes to `Sorter` that reduce unnecessary work.
Makes TPC-H query 1 roughly 2x faster, which is a lot because it
originally took 30-40 seconds depending on the CI run
Closes#3645
I found an application in the open that expects sqlite_version() to
return a specific string (higher than 3.8...).
We had tons of those issues at Scylla, and the lesson was that you tell
your kids not to lie, but when life hits, well... you lie.
We'll add a new function, turso_version, that tells the truth.
Closes#3635
