aljaz/turso
1
0
Fork 0
mirror of https://github.com/aljazceru/turso.git synced 2025-12-17 08:34:19 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge 'Simulator: refactor and simplify InteractionPlan' from Pedro Muniz

Browse Source 
Depends on #3775 - to remove noise from this PR.
## Motivation
In my continued efforts in making the simulator more accessible and
simpler to work with, I have over time simplified and optimized some
parts of the codebase like query generation and decision making so that
more people from the community can contribute and enhance the simulator.
This PR is one more step in that direction.
Before this PR, our `InteractionPlan` stored `Vec<Interactions>`.
`Interactions` are a higher level collection that will generate a list
of `Interaction` (yes I know the naming can be slightly confusing
sometimes. Maybe we can change it later as well. Especially because
`Interactions` are mainly just `Property`). However, this architecture
imposed a problem when MVCC enters the picture. MVCC requires us to make
sure that DDL statements are executed serially. To avoid adding even
more complexity to plan generation, I opted on previous PRs to check
before emitting an `Interaction` for execution, if the interaction is a
DDL statement, and if it is, I emit a `Commit` for each connection still
in a transaction. This worked slightly fine, but as we do not store the
actual execution of interactions in the interaction plan, only the
higher level `Interactions`, this meant that I had to do some
workarounds to modify the `Interactions` inside the plan to persist the
`Commit` I generated on demand.
## Problem
However, I was stupid and overlooked the fact that for certain
properties that allow queries to be generated in the middle (referenced
as extensional queries in the code), we cannot specify the connection
that should execute that query, meaning if a DDL statement occurred
there, the simulator could emit the query but could not save it properly
in the plan to reproduce in shrinking. So to correct and make
interaction generation/emission less brittle, I refactored the
`InteractionPlan` so that it stores `Vec<Interaction>` instead.
## Implications
- `Interaction` is not currently serializable using `Serde` due to the
fact that it stores a function in `Assertion`. This means that we cannot
serialize the plan into a `plan.json`. Which to me is honestly fine, as
the only things that used `plan.json` was `--load` and `--watch`
options. Which are options almost nobody really used.
- For load, instead of generating the whole plan it just read the plan
from disk. The workaround for that right now is just load the `cli_opts`
that were last run for that particular seed and use those exact options
to run the simulation.
- For watch, currently there is not workaround but, @alpaylan told me
has some plans to make assertions serializable by embedding a custom
language into the `plan.sql` file, meaning we will probably not need a
json file at all to store the interaction plan. And this embedded
language will make it much easier to bring back a more proper watch
mode.
- The current shrinking algorithms all have some notion of properties
and removal of properties, but `Interaction` do not have this concept.
So I added some metadata to interactions and a origin ID to each
`Interaction` so that we can search through the list of interactions
using binary search to get all of the interactions that are part of the
same `Property`. To support this, I added an `InteractionBuilder` and
some utilities to iterate and remove properties in the `InteractionPlan`
## Conclusion
Overall, this code simplifies emission of interactions and ensures the
`InteractionPlan` always stores the actual interactions that get
executed. This also decouples more query generation logic from query
emission logic.

Closes #3774
This commit is contained in:
Jussi Saurio
2025-11-19 11:10:51 +02:00
committed by GitHub
parent 687d9faf37 c2be60b007
commit fb31fd56ba
17 changed files with 2058 additions and 2232 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
Cargo.lock generated
View File

@@ -1225,6 +1225,37 @@ dependencies = [
"powerfmt",
]
[[package]]
name = "derive_builder"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947"
dependencies = [
"derive_builder_macro",
]
[[package]]
name = "derive_builder_core"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8"
dependencies = [
"darling",
"proc-macro2",
"quote",
"syn 2.0.100",
]
[[package]]
name = "derive_builder_macro"
version = "0.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c"
dependencies = [
"derive_builder_core",
"syn 2.0.100",
]
[[package]]
name = "derive_more"
version = "2.0.1"
@@ -2617,6 +2648,7 @@ dependencies = [
"bitmaps",
"chrono",
"clap",
"derive_builder",
"dirs 6.0.0",
"either",
"garde",

1
simulator/Cargo.toml
View File

@@ -47,3 +47,4 @@ similar = { workspace = true }
similar-asserts = { workspace = true }
bitmaps = { workspace = true }
bitflags.workspace = true
derive_builder = "0.20.2"

1229
simulator/generation/plan.rs
View File

File diff suppressed because it is too large Load Diff

551
simulator/generation/property.rs
View File

@@ -5,8 +5,9 @@
//! we can generate queries that reference tables that do not exist. This is not a correctness issue, but more of
//! an optimization issue that is good to point out for the future
use std::num::NonZeroUsize;
use rand::distr::{Distribution, weighted::WeightedIndex};
use serde::{Deserialize, Serialize};
use sql_generation::{
generation::{Arbitrary, ArbitraryFrom, GenerationContext, pick, pick_index},
model::{
@@ -27,253 +28,22 @@ use turso_parser::ast::{self, Distinctness};
use crate::{
common::print_diff,
generation::{
Shadow as _, WeightedDistribution, plan::InteractionType, query::QueryDistribution,
generation::{Shadow, WeightedDistribution, query::QueryDistribution},
model::{
Query, QueryCapabilities, QueryDiscriminants, ResultSet,
interactions::{
Assertion, Interaction, InteractionBuilder, InteractionType, PropertyMetadata,
},
metrics::Remaining,
property::{InteractiveQueryInfo, Property, PropertyDiscriminants},
},
model::{Query, QueryCapabilities, QueryDiscriminants},
profiles::query::QueryProfile,
runner::env::SimulatorEnv,
};
use super::plan::{Assertion, Interaction, InteractionStats, ResultSet};
/// Properties are representations of executable specifications
/// about the database behavior.
#[derive(Debug, Clone, Serialize, Deserialize, strum::EnumDiscriminants)]
#[strum_discriminants(derive(strum::EnumIter))]
pub enum Property {
/// Insert-Select is a property in which the inserted row
/// must be in the resulting rows of a select query that has a
/// where clause that matches the inserted row.
/// The execution of the property is as follows
/// INSERT INTO <t> VALUES (...)
/// I_0
/// I_1
/// ...
/// I_n
/// SELECT * FROM <t> WHERE <predicate>
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - The inserted row will not be deleted.
/// - The inserted row will not be updated.
/// - The table `t` will not be renamed, dropped, or altered.
InsertValuesSelect {
/// The insert query
insert: Insert,
/// Selected row index
row_index: usize,
/// Additional interactions in the middle of the property
queries: Vec<Query>,
/// The select query
select: Select,
/// Interactive query information if any
interactive: Option<InteractiveQueryInfo>,
},
/// ReadYourUpdatesBack is a property in which the updated rows
/// must be in the resulting rows of a select query that has a
/// where clause that matches the updated row.
/// The execution of the property is as follows
/// UPDATE <t> SET <set_cols=set_vals> WHERE <predicate>
/// SELECT <set_cols> FROM <t> WHERE <predicate>
/// These interactions are executed in immediate succession
/// just to verify the property that our updates did what they
/// were supposed to do.
ReadYourUpdatesBack {
update: Update,
select: Select,
},
/// TableHasExpectedContent is a property in which the table
/// must have the expected content, i.e. all the insertions and
/// updates and deletions should have been persisted in the way
/// we think they were.
/// The execution of the property is as follows
/// SELECT * FROM <t>
/// ASSERT <expected_content>
TableHasExpectedContent {
table: String,
},
/// AllTablesHaveExpectedContent is a property in which the table
/// must have the expected content, i.e. all the insertions and
/// updates and deletions should have been persisted in the way
/// we think they were.
/// The execution of the property is as follows
/// SELECT * FROM <t>
/// ASSERT <expected_content>
/// for each table in the simulator model
AllTableHaveExpectedContent {
tables: Vec<String>,
},
/// Double Create Failure is a property in which creating
/// the same table twice leads to an error.
/// The execution of the property is as follows
/// CREATE TABLE <t> (...)
/// I_0
/// I_1
/// ...
/// I_n
/// CREATE TABLE <t> (...) -> Error
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - Table `t` will not be renamed or dropped.
DoubleCreateFailure {
/// The create query
create: Create,
/// Additional interactions in the middle of the property
queries: Vec<Query>,
},
/// Select Limit is a property in which the select query
/// has a limit clause that is respected by the query.
/// The execution of the property is as follows
/// SELECT * FROM <t> WHERE <predicate> LIMIT <n>
/// This property is a single-interaction property.
/// The interaction has the following constraints;
/// - The select query will respect the limit clause.
SelectLimit {
/// The select query
select: Select,
},
/// Delete-Select is a property in which the deleted row
/// must not be in the resulting rows of a select query that has a
/// where clause that matches the deleted row. In practice, `p1` of
/// the delete query will be used as the predicate for the select query,
/// hence the select should return NO ROWS.
/// The execution of the property is as follows
/// DELETE FROM <t> WHERE <predicate>
/// I_0
/// I_1
/// ...
/// I_n
/// SELECT * FROM <t> WHERE <predicate>
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - A row that holds for the predicate will not be inserted.
/// - The table `t` will not be renamed, dropped, or altered.
DeleteSelect {
table: String,
predicate: Predicate,
queries: Vec<Query>,
},
/// Drop-Select is a property in which selecting from a dropped table
/// should result in an error.
/// The execution of the property is as follows
/// DROP TABLE <t>
/// I_0
/// I_1
/// ...
/// I_n
/// SELECT * FROM <t> WHERE <predicate> -> Error
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - The table `t` will not be created, no table will be renamed to `t`.
DropSelect {
table: String,
queries: Vec<Query>,
select: Select,
},
/// Select-Select-Optimizer is a property in which we test the optimizer by
/// running two equivalent select queries, one with `SELECT <predicate> from <t>`
/// and the other with `SELECT * from <t> WHERE <predicate>`. As highlighted by
/// Rigger et al. in Non-Optimizing Reference Engine Construction(NoREC), SQLite
/// tends to optimize `where` statements while keeping the result column expressions
/// unoptimized. This property is used to test the optimizer. The property is successful
/// if the two queries return the same number of rows.
SelectSelectOptimizer {
table: String,
predicate: Predicate,
},
/// Where-True-False-Null is a property that tests the boolean logic implementation
/// in the database. It relies on the fact that `P == true || P == false || P == null` should return true,
/// as SQLite uses a ternary logic system. This property is invented in "Finding Bugs in Database Systems via Query Partitioning"
/// by Rigger et al. and it is canonically called Ternary Logic Partitioning (TLP).
WhereTrueFalseNull {
select: Select,
predicate: Predicate,
},
/// UNION-ALL-Preserves-Cardinality is a property that tests the UNION ALL operator
/// implementation in the database. It relies on the fact that `SELECT * FROM <t
/// > WHERE <predicate> UNION ALL SELECT * FROM <t> WHERE <predicate>`
/// should return the same number of rows as `SELECT <predicate> FROM <t> WHERE <predicate>`.
/// > The property is succesfull when the UNION ALL of 2 select queries returns the same number of rows
/// > as the sum of the two select queries.
UNIONAllPreservesCardinality {
select: Select,
where_clause: Predicate,
},
/// FsyncNoWait is a property which tests if we do not loose any data after not waiting for fsync.
///
/// # Interactions
/// - Executes the `query` without waiting for fsync
/// - Drop all connections and Reopen the database
/// - Execute the `query` again
/// - Query tables to assert that the values were inserted
///
FsyncNoWait {
query: Query,
},
FaultyQuery {
query: Query,
},
/// Property used to subsititute a property with its queries only
Queries {
queries: Vec<Query>,
},
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct InteractiveQueryInfo {
start_with_immediate: bool,
end_with_commit: bool,
}
type PropertyQueryGenFunc<'a, R, G> =
fn(&mut R, &G, &QueryDistribution, &Property) -> Option<Query>;
impl Property {
pub(crate) fn name(&self) -> &str {
match self {
Property::InsertValuesSelect { .. } => "Insert-Values-Select",
Property::ReadYourUpdatesBack { .. } => "Read-Your-Updates-Back",
Property::TableHasExpectedContent { .. } => "Table-Has-Expected-Content",
Property::AllTableHaveExpectedContent { .. } => "All-Tables-Have-Expected-Content",
Property::DoubleCreateFailure { .. } => "Double-Create-Failure",
Property::SelectLimit { .. } => "Select-Limit",
Property::DeleteSelect { .. } => "Delete-Select",
Property::DropSelect { .. } => "Drop-Select",
Property::SelectSelectOptimizer { .. } => "Select-Select-Optimizer",
Property::WhereTrueFalseNull { .. } => "Where-True-False-Null",
Property::FsyncNoWait { .. } => "FsyncNoWait",
Property::FaultyQuery { .. } => "FaultyQuery",
Property::UNIONAllPreservesCardinality { .. } => "UNION-All-Preserves-Cardinality",
Property::Queries { .. } => "Queries",
}
}
/// Property Does some sort of fault injection
pub fn check_tables(&self) -> bool {
matches!(
self,
Property::FsyncNoWait { .. } | Property::FaultyQuery { .. }
)
}
pub fn get_extensional_queries(&mut self) -> Option<&mut Vec<Query>> {
match self {
Property::InsertValuesSelect { queries, .. }
| Property::DoubleCreateFailure { queries, .. }
| Property::DeleteSelect { queries, .. }
| Property::DropSelect { queries, .. }
| Property::Queries { queries } => Some(queries),
Property::FsyncNoWait { .. } | Property::FaultyQuery { .. } => None,
Property::SelectLimit { .. }
| Property::SelectSelectOptimizer { .. }
| Property::WhereTrueFalseNull { .. }
| Property::UNIONAllPreservesCardinality { .. }
| Property::ReadYourUpdatesBack { .. }
| Property::TableHasExpectedContent { .. }
| Property::AllTableHaveExpectedContent { .. } => None,
}
}
pub(super) fn get_extensional_query_gen_function<R, G>(&self) -> PropertyQueryGenFunc<R, G>
where
R: rand::Rng + ?Sized,
@@ -465,7 +235,7 @@ impl Property {
Property::SelectLimit { .. }
| Property::SelectSelectOptimizer { .. }
| Property::WhereTrueFalseNull { .. }
| Property::UNIONAllPreservesCardinality { .. }
| Property::UnionAllPreservesCardinality { .. }
| Property::ReadYourUpdatesBack { .. }
| Property::TableHasExpectedContent { .. }
| Property::AllTableHaveExpectedContent { .. } => {
@@ -477,13 +247,18 @@ impl Property {
/// interactions construct a list of interactions, which is an executable representation of the property.
/// the requirement of property -> vec<interaction> conversion emerges from the need to serialize the property,
/// and `interaction` cannot be serialized directly.
pub(crate) fn interactions(&self, connection_index: usize) -> Vec<Interaction> {
match self {
pub(crate) fn interactions(
&self,
connection_index: usize,
id: NonZeroUsize,
) -> Vec<Interaction> {
let interactions: Vec<InteractionBuilder> = match self {
Property::AllTableHaveExpectedContent { tables } => {
assert_all_table_values(tables, connection_index).collect()
}
Property::TableHasExpectedContent { table } => {
let table = table.to_string();
let table_dependency = table.clone();
let table_name = table.clone();
let assumption = InteractionType::Assumption(Assertion::new(
format!("table {} exists", table.clone()),
@@ -495,6 +270,7 @@ impl Property {
Ok(Err(format!("table {table_name} does not exist")))
}
},
vec![table_dependency.clone()],
));
let select_interaction = InteractionType::Query(Query::Select(Select::simple(
@@ -535,16 +311,18 @@ impl Property {
}
Ok(Ok(()))
},
vec![table_dependency.clone()],
));
vec![
Interaction::new(connection_index, assumption),
Interaction::new(connection_index, select_interaction),
Interaction::new(connection_index, assertion),
InteractionBuilder::with_interaction(assumption),
InteractionBuilder::with_interaction(select_interaction),
InteractionBuilder::with_interaction(assertion),
]
}
Property::ReadYourUpdatesBack { update, select } => {
let table = update.table().to_string();
let table_dependency = table.clone();
let assumption = InteractionType::Assumption(Assertion::new(
format!("table {} exists", table.clone()),
move |_: &Vec<ResultSet>, env: &mut SimulatorEnv| {
@@ -555,6 +333,7 @@ impl Property {
Ok(Err(format!("table {} does not exist", table.clone())))
}
},
vec![table_dependency.clone()],
));
let update_interaction = InteractionType::Query(Query::Update(update.clone()));
@@ -599,13 +378,14 @@ impl Property {
Err(err) => Err(LimboError::InternalError(err.to_string())),
}
},
vec![table_dependency],
));
vec![
Interaction::new(connection_index, assumption),
Interaction::new(connection_index, update_interaction),
Interaction::new(connection_index, select_interaction),
Interaction::new(connection_index, assertion),
InteractionBuilder::with_interaction(assumption),
InteractionBuilder::with_interaction(update_interaction),
InteractionBuilder::with_interaction(select_interaction),
InteractionBuilder::with_interaction(assertion),
]
}
Property::InsertValuesSelect {
@@ -645,6 +425,7 @@ impl Property {
}
}
},
vec![insert.table().to_string()],
));
let assertion = InteractionType::Assertion(Assertion::new(
@@ -679,30 +460,30 @@ impl Property {
Err(err) => Err(LimboError::InternalError(err.to_string())),
}
},
vec![insert.table().to_string()],
));
let mut interactions = Vec::new();
interactions.push(Interaction::new(connection_index, assumption));
interactions.push(Interaction::new(
connection_index,
interactions.push(InteractionBuilder::with_interaction(assumption));
interactions.push(InteractionBuilder::with_interaction(
InteractionType::Query(Query::Insert(insert.clone())),
));
interactions.extend(
queries
.clone()
.into_iter()
.map(|q| Interaction::new(connection_index, InteractionType::Query(q))),
);
interactions.push(Interaction::new(
connection_index,
interactions.extend(queries.clone().into_iter().map(|q| {
let mut builder =
InteractionBuilder::with_interaction(InteractionType::Query(q));
builder.property_meta(PropertyMetadata::new(self, true));
builder
}));
interactions.push(InteractionBuilder::with_interaction(
InteractionType::Query(Query::Select(select.clone())),
));
interactions.push(Interaction::new(connection_index, assertion));
interactions.push(InteractionBuilder::with_interaction(assertion));
interactions
}
Property::DoubleCreateFailure { create, queries } => {
let table_name = create.table.name.clone();
let table_dependency = table_name.clone();
let assumption = InteractionType::Assumption(Assertion::new(
"Double-Create-Failure should not be called on an existing table".to_string(),
@@ -714,6 +495,7 @@ impl Property {
Ok(Err(format!("table {table_name} already exists")))
}
},
vec![table_dependency.clone()],
));
let cq1 = InteractionType::Query(Query::Create(create.clone()));
@@ -736,19 +518,23 @@ impl Property {
}
}
}
}) );
}, vec![table_dependency],) );
let mut interactions = Vec::new();
interactions.push(Interaction::new(connection_index, assumption));
interactions.push(Interaction::new(connection_index, cq1));
interactions.extend(
queries
.clone()
.into_iter()
.map(|q| Interaction::new(connection_index, InteractionType::Query(q))),
);
interactions.push(Interaction::new_ignore_error(connection_index, cq2));
interactions.push(Interaction::new(connection_index, assertion));
interactions.push(InteractionBuilder::with_interaction(assumption));
interactions.push(InteractionBuilder::with_interaction(cq1));
interactions.extend(queries.clone().into_iter().map(|q| {
let mut builder =
InteractionBuilder::with_interaction(InteractionType::Query(q));
builder.property_meta(PropertyMetadata::new(self, true));
builder
}));
interactions.push({
let mut builder = InteractionBuilder::with_interaction(cq2);
builder.ignore_error(true);
builder
});
interactions.push(InteractionBuilder::with_interaction(assertion));
interactions
}
@@ -780,6 +566,7 @@ impl Property {
}
}
},
select.dependencies().into_iter().collect(),
));
let limit = select
@@ -805,15 +592,15 @@ impl Property {
Err(_) => Ok(Ok(())),
}
},
select.dependencies().into_iter().collect(),
));
vec![
Interaction::new(connection_index, assumption),
Interaction::new(
connection_index,
InteractionType::Query(Query::Select(select.clone())),
),
Interaction::new(connection_index, assertion),
InteractionBuilder::with_interaction(assumption),
InteractionBuilder::with_interaction(InteractionType::Query(Query::Select(
select.clone(),
))),
InteractionBuilder::with_interaction(assertion),
]
}
Property::DeleteSelect {
@@ -840,6 +627,7 @@ impl Property {
}
}
},
vec![table.clone()],
));
let delete = InteractionType::Query(Query::Delete(Delete {
@@ -874,19 +662,20 @@ impl Property {
Err(err) => Err(LimboError::InternalError(err.to_string())),
}
},
vec![table.clone()],
));
let mut interactions = Vec::new();
interactions.push(Interaction::new(connection_index, assumption));
interactions.push(Interaction::new(connection_index, delete));
interactions.extend(
queries
.clone()
.into_iter()
.map(|q| Interaction::new(connection_index, InteractionType::Query(q))),
);
interactions.push(Interaction::new(connection_index, select));
interactions.push(Interaction::new(connection_index, assertion));
interactions.push(InteractionBuilder::with_interaction(assumption));
interactions.push(InteractionBuilder::with_interaction(delete));
interactions.extend(queries.clone().into_iter().map(|q| {
let mut builder =
InteractionBuilder::with_interaction(InteractionType::Query(q));
builder.property_meta(PropertyMetadata::new(self, true));
builder
}));
interactions.push(InteractionBuilder::with_interaction(select));
interactions.push(InteractionBuilder::with_interaction(assertion));
interactions
}
@@ -914,6 +703,7 @@ impl Property {
}
}
},
vec![table.clone()],
));
let table_name = table.clone();
@@ -939,6 +729,7 @@ impl Property {
},
}
},
vec![table.clone()],
));
let drop = InteractionType::Query(Query::Drop(Drop {
@@ -949,16 +740,20 @@ impl Property {
let mut interactions = Vec::new();
interactions.push(Interaction::new(connection_index, assumption));
interactions.push(Interaction::new(connection_index, drop));
interactions.extend(
queries
.clone()
.into_iter()
.map(|q| Interaction::new(connection_index, InteractionType::Query(q))),
);
interactions.push(Interaction::new_ignore_error(connection_index, select));
interactions.push(Interaction::new(connection_index, assertion));
interactions.push(InteractionBuilder::with_interaction(assumption));
interactions.push(InteractionBuilder::with_interaction(drop));
interactions.extend(queries.clone().into_iter().map(|q| {
let mut builder =
InteractionBuilder::with_interaction(InteractionType::Query(q));
builder.property_meta(PropertyMetadata::new(self, true));
builder
}));
interactions.push({
let mut builder = InteractionBuilder::with_interaction(select);
builder.ignore_error(true);
builder
});
interactions.push(InteractionBuilder::with_interaction(assertion));
interactions
}
@@ -982,6 +777,7 @@ impl Property {
}
}
},
vec![table.clone()],
));
let select1 = InteractionType::Query(Query::Select(Select::single(
@@ -1042,18 +838,18 @@ impl Property {
}
}
},
vec![table.clone()],
));
vec![
Interaction::new(connection_index, assumption),
Interaction::new(connection_index, select1),
Interaction::new(connection_index, select2),
Interaction::new(connection_index, assertion),
InteractionBuilder::with_interaction(assumption),
InteractionBuilder::with_interaction(select1),
InteractionBuilder::with_interaction(select2),
InteractionBuilder::with_interaction(assertion),
]
}
Property::FsyncNoWait { query } => {
vec![Interaction::new(
connection_index,
vec![InteractionBuilder::with_interaction(
InteractionType::FsyncQuery(query.clone()),
)]
}
@@ -1083,16 +879,18 @@ impl Property {
}
}
},
query.dependencies().into_iter().collect(),
);
[
InteractionType::FaultyQuery(query.clone()),
InteractionType::Assertion(assert),
]
.into_iter()
.map(|i| Interaction::new(connection_index, i))
.map(InteractionBuilder::with_interaction)
.collect()
}
Property::WhereTrueFalseNull { select, predicate } => {
let tables_dependencies = select.dependencies().into_iter().collect::<Vec<_>>();
let assumption = InteractionType::Assumption(Assertion::new(
format!(
"tables ({}) exists",
@@ -1120,6 +918,7 @@ impl Property {
}
}
},
tables_dependencies.clone(),
));
let old_predicate = select.body.select.where_clause.clone();
@@ -1241,16 +1040,17 @@ impl Property {
}
}
},
tables_dependencies,
));
vec![
Interaction::new(connection_index, assumption),
Interaction::new(connection_index, select),
Interaction::new(connection_index, select_tlp),
Interaction::new(connection_index, assertion),
InteractionBuilder::with_interaction(assumption),
InteractionBuilder::with_interaction(select),
InteractionBuilder::with_interaction(select_tlp),
InteractionBuilder::with_interaction(assertion),
]
}
Property::UNIONAllPreservesCardinality {
Property::UnionAllPreservesCardinality {
select,
where_clause,
} => {
@@ -1295,22 +1095,37 @@ impl Property {
}
}
},
)),
].into_iter().map(|i| Interaction::new(connection_index, i)).collect()
s3.dependencies().into_iter().collect()
)
),
].into_iter().map(InteractionBuilder::with_interaction).collect()
}
Property::Queries { queries } => queries
.clone()
.into_iter()
.map(|query| Interaction::new(connection_index, InteractionType::Query(query)))
.map(|query| InteractionBuilder::with_interaction(InteractionType::Query(query)))
.collect(),
};
assert!(!interactions.is_empty());
interactions
.into_iter()
.map(|mut builder| {
if !builder.has_property_meta() {
builder.property_meta(PropertyMetadata::new(self, false));
}
builder.connection_index(connection_index).id(id);
builder.build().unwrap()
})
.collect()
}
}
fn assert_all_table_values(
tables: &[String],
connection_index: usize,
) -> impl Iterator<Item = Interaction> + use<'_> {
) -> impl Iterator<Item = InteractionBuilder> + use<'_> {
tables.iter().flat_map(move |table| {
let select = InteractionType::Query(Query::Select(Select::simple(
table.clone(),
@@ -1364,105 +1179,11 @@ fn assert_all_table_values(
Err(err) => Err(LimboError::InternalError(format!("{err}"))),
}
}
}));
[select, assertion].into_iter().map(move |i| Interaction::new(connection_index, i))
}, vec![table.clone()]));
[select, assertion].into_iter().map(InteractionBuilder::with_interaction)
})
}
#[derive(Debug)]
pub(super) struct Remaining {
pub select: u32,
pub insert: u32,
pub create: u32,
pub create_index: u32,
pub delete: u32,
pub update: u32,
pub drop: u32,
pub alter_table: u32,
pub drop_index: u32,
pub pragma_count: u32,
}
pub(super) fn remaining(
max_interactions: u32,
opts: &QueryProfile,
stats: &InteractionStats,
mvcc: bool,
context: &impl GenerationContext,
) -> Remaining {
let total_weight = opts.total_weight();
let total_select = (max_interactions * opts.select_weight) / total_weight;
let total_insert = (max_interactions * opts.insert_weight) / total_weight;
let total_create = (max_interactions * opts.create_table_weight) / total_weight;
let total_create_index = (max_interactions * opts.create_index_weight) / total_weight;
let total_delete = (max_interactions * opts.delete_weight) / total_weight;
let total_update = (max_interactions * opts.update_weight) / total_weight;
let total_drop = (max_interactions * opts.drop_table_weight) / total_weight;
let total_alter_table = (max_interactions * opts.alter_table_weight) / total_weight;
let total_drop_index = (max_interactions * opts.drop_index) / total_weight;
let total_pragma = (max_interactions * opts.pragma_weight) / total_weight;
let remaining_select = total_select
.checked_sub(stats.select_count)
.unwrap_or_default();
let remaining_insert = total_insert
.checked_sub(stats.insert_count)
.unwrap_or_default();
let remaining_create = total_create
.checked_sub(stats.create_count)
.unwrap_or_default();
let mut remaining_create_index = total_create_index
.checked_sub(stats.create_index_count)
.unwrap_or_default();
let remaining_delete = total_delete
.checked_sub(stats.delete_count)
.unwrap_or_default();
let remaining_update = total_update
.checked_sub(stats.update_count)
.unwrap_or_default();
let remaining_drop = total_drop.checked_sub(stats.drop_count).unwrap_or_default();
let remaining_pragma = total_pragma
.checked_sub(stats.pragma_count)
.unwrap_or_default();
let remaining_alter_table = total_alter_table
.checked_sub(stats.alter_table_count)
.unwrap_or_default();
let mut remaining_drop_index = total_drop_index
.checked_sub(stats.alter_table_count)
.unwrap_or_default();
if mvcc {
// TODO: index not supported yet for mvcc
remaining_create_index = 0;
remaining_drop_index = 0;
}
// if there are no indexes do not allow creation of drop_index
if !context
.tables()
.iter()
.any(|table| !table.indexes.is_empty())
{
remaining_drop_index = 0;
}
Remaining {
select: remaining_select,
insert: remaining_insert,
create: remaining_create,
create_index: remaining_create_index,
delete: remaining_delete,
drop: remaining_drop,
update: remaining_update,
alter_table: remaining_alter_table,
drop_index: remaining_drop_index,
pragma_count: remaining_pragma,
}
}
fn property_insert_values_select<R: rand::Rng + ?Sized>(
rng: &mut R,
_query_distr: &QueryDistribution,
@@ -1739,7 +1460,7 @@ fn property_union_all_preserves_cardinality<R: rand::Rng + ?Sized>(
Distinctness::All,
);
Property::UNIONAllPreservesCardinality {
Property::UnionAllPreservesCardinality {
select,
where_clause: p2,
}
@@ -1788,7 +1509,7 @@ impl PropertyDiscriminants {
PropertyDiscriminants::DropSelect => property_drop_select,
PropertyDiscriminants::SelectSelectOptimizer => property_select_select_optimizer,
PropertyDiscriminants::WhereTrueFalseNull => property_where_true_false_null,
PropertyDiscriminants::UNIONAllPreservesCardinality => {
PropertyDiscriminants::UnionAllPreservesCardinality => {
property_union_all_preserves_cardinality
}
PropertyDiscriminants::FsyncNoWait => property_fsync_no_wait,
@@ -1871,7 +1592,7 @@ impl PropertyDiscriminants {
0
}
}
PropertyDiscriminants::UNIONAllPreservesCardinality => {
PropertyDiscriminants::UnionAllPreservesCardinality => {
if opts.indexes
&& !env.opts.disable_union_all_preserves_cardinality
&& !ctx.tables().is_empty()
@@ -1935,7 +1656,7 @@ impl PropertyDiscriminants {
}
PropertyDiscriminants::SelectSelectOptimizer => QueryCapabilities::SELECT,
PropertyDiscriminants::WhereTrueFalseNull => QueryCapabilities::SELECT,
PropertyDiscriminants::UNIONAllPreservesCardinality => QueryCapabilities::SELECT,
PropertyDiscriminants::UnionAllPreservesCardinality => QueryCapabilities::SELECT,
PropertyDiscriminants::FsyncNoWait => QueryCapabilities::all(),
PropertyDiscriminants::FaultyQuery => QueryCapabilities::all(),
PropertyDiscriminants::Queries => panic!("queries property should not be generated"),

4
simulator/generation/query.rs
View File

@@ -1,6 +1,6 @@
use crate::{
generation::WeightedDistribution,
model::{Query, QueryDiscriminants},
model::{Query, QueryDiscriminants, metrics::Remaining},
};
use rand::{
Rng,
@@ -20,8 +20,6 @@ use sql_generation::{
},
};
use super::property::Remaining;
fn random_create<R: rand::Rng + ?Sized>(rng: &mut R, conn_ctx: &impl GenerationContext) -> Query {
let mut create = Create::arbitrary(rng, conn_ctx);
while conn_ctx

149
simulator/main.rs
View File

@@ -1,11 +1,8 @@
#![allow(clippy::arc_with_non_send_sync)]
use anyhow::anyhow;
use clap::Parser;
use generation::plan::{InteractionPlan, InteractionPlanState};
use notify::event::{DataChange, ModifyKind};
use notify::{EventKind, RecursiveMode, Watcher};
use rand::prelude::*;
use runner::bugbase::{Bug, BugBase, LoadedBug};
use runner::bugbase::BugBase;
use runner::cli::{SimulatorCLI, SimulatorCommand};
use runner::differential;
use runner::env::SimulatorEnv;
@@ -16,13 +13,15 @@ use std::fs::OpenOptions;
use std::io::{IsTerminal, Write};
use std::path::Path;
use std::rc::Rc;
use std::sync::{Arc, Mutex, mpsc};
use std::sync::{Arc, Mutex};
use tracing_subscriber::EnvFilter;
use tracing_subscriber::field::MakeExt;
use tracing_subscriber::fmt::format;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
use crate::generation::plan::{ConnectionState, InteractionPlanIterator};
use crate::model::interactions::{
ConnectionState, InteractionPlan, InteractionPlanIterator, InteractionPlanState,
};
use crate::profiles::Profile;
use crate::runner::doublecheck;
use crate::runner::env::{Paths, SimulationPhase, SimulationType};
@@ -42,7 +41,7 @@ fn main() -> anyhow::Result<()> {
let profile = Profile::parse_from_type(cli_opts.profile.clone())?;
tracing::debug!(sim_profile = ?profile);
if let Some(ref command) = cli_opts.subcommand {
if let Some(command) = cli_opts.subcommand.take() {
match command {
SimulatorCommand::List => {
let mut bugbase = BugBase::load()?;
@@ -50,10 +49,10 @@ fn main() -> anyhow::Result<()> {
}
SimulatorCommand::Loop { n, short_circuit } => {
banner();
for i in 0..*n {
for i in 0..n {
println!("iteration {i}");
let result = testing_main(&cli_opts, &profile);
if result.is_err() && *short_circuit {
let result = testing_main(&mut cli_opts, &profile);
if result.is_err() && short_circuit {
println!("short circuiting after {i} iterations");
return result;
} else if result.is_err() {
@@ -65,7 +64,7 @@ fn main() -> anyhow::Result<()> {
Ok(())
}
SimulatorCommand::Test { filter } => {
let mut bugbase = BugBase::load()?;
let bugbase = BugBase::load()?;
let bugs = bugbase.load_bugs()?;
let mut bugs = bugs
.into_iter()
@@ -74,7 +73,7 @@ fn main() -> anyhow::Result<()> {
.runs
.into_iter()
.filter_map(|run| run.error.clone().map(|_| run))
.filter(|run| run.error.as_ref().unwrap().contains(filter))
.filter(|run| run.error.as_ref().unwrap().contains(&filter))
.map(|run| run.cli_options)
.collect::<Vec<_>>();
@@ -99,7 +98,7 @@ fn main() -> anyhow::Result<()> {
let results = bugs
.into_iter()
.map(|cli_opts| testing_main(&cli_opts, &profile))
.map(|mut cli_opts| testing_main(&mut cli_opts, &profile))
.collect::<Vec<_>>();
let (successes, failures): (Vec<_>, Vec<_>) =
@@ -117,11 +116,11 @@ fn main() -> anyhow::Result<()> {
}
} else {
banner();
testing_main(&cli_opts, &profile)
testing_main(&mut cli_opts, &profile)
}
}
fn testing_main(cli_opts: &SimulatorCLI, profile: &Profile) -> anyhow::Result<()> {
fn testing_main(cli_opts: &mut SimulatorCLI, profile: &Profile) -> anyhow::Result<()> {
let mut bugbase = if cli_opts.disable_bugbase {
None
} else {
@@ -132,8 +131,7 @@ fn testing_main(cli_opts: &SimulatorCLI, profile: &Profile) -> anyhow::Result<()
let (seed, mut env, plans) = setup_simulation(bugbase.as_mut(), cli_opts, profile);
if cli_opts.watch {
watch_mode(env).unwrap();
return Ok(());
anyhow::bail!("watch mode is disabled for now");
}
let paths = env.paths.clone();
@@ -157,58 +155,6 @@ fn testing_main(cli_opts: &SimulatorCLI, profile: &Profile) -> anyhow::Result<()
result
}
fn watch_mode(env: SimulatorEnv) -> notify::Result<()> {
let (tx, rx) = mpsc::channel::<notify::Result<notify::Event>>();
println!("watching {:?}", env.get_plan_path());
// Use recommended_watcher() to automatically select the best implementation
// for your platform. The `EventHandler` passed to this constructor can be a
// closure, a `std::sync::mpsc::Sender`, a `crossbeam_channel::Sender`, or
// another type the trait is implemented for.
let mut watcher = notify::recommended_watcher(tx)?;
// Add a path to be watched. All files and directories at that path and
// below will be monitored for changes.
watcher.watch(&env.get_plan_path(), RecursiveMode::NonRecursive)?;
// Block forever, printing out events as they come in
let last_execution = Arc::new(Mutex::new(Execution::new(0, 0)));
for res in rx {
match res {
Ok(event) => {
if let EventKind::Modify(ModifyKind::Data(DataChange::Content)) = event.kind {
tracing::info!("plan file modified, rerunning simulation");
let env = env.clone_without_connections();
let last_execution_ = last_execution.clone();
let result = SandboxedResult::from(
std::panic::catch_unwind(move || {
let mut env = env;
let plan_path = env.get_plan_path();
let plan = InteractionPlan::compute_via_diff(&plan_path);
env.clear();
let env = Arc::new(Mutex::new(env.clone_without_connections()));
run_simulation_default(env, plan, last_execution_.clone())
}),
last_execution.clone(),
);
match result {
SandboxedResult::Correct => {
tracing::info!("simulation succeeded");
println!("simulation succeeded");
}
SandboxedResult::Panicked { error, .. }
| SandboxedResult::FoundBug { error, .. } => {
tracing::error!("simulation failed: '{}'", error);
}
}
}
}
Err(e) => println!("watch error: {e:?}"),
}
}
Ok(())
}
fn run_simulator(
mut bugbase: Option<&mut BugBase>,
cli_opts: &SimulatorCLI,
@@ -259,11 +205,6 @@ fn run_simulator(
tracing::info!("{}", plan.stats());
std::fs::write(env.get_plan_path(), plan.to_string()).unwrap();
std::fs::write(
env.get_plan_path().with_extension("json"),
serde_json::to_string_pretty(&*plan).unwrap(),
)
.unwrap();
// No doublecheck, run shrinking if panicking or found a bug.
match &result {
@@ -348,7 +289,7 @@ fn run_simulator(
tracing::trace!(
"adding bug to bugbase, seed: {}, plan: {}, error: {}",
env.opts.seed,
plan.len(),
plan.len_properties(),
error
);
bugbase
@@ -384,7 +325,7 @@ fn run_simulator(
);
// Save the shrunk database
if let Some(bugbase) = bugbase.as_deref_mut() {
bugbase.make_shrunk(
bugbase.save_shrunk(
seed,
cli_opts,
final_plan.clone(),
@@ -470,55 +411,34 @@ impl SandboxedResult {
}
fn setup_simulation(
bugbase: Option<&mut BugBase>,
cli_opts: &SimulatorCLI,
mut bugbase: Option<&mut BugBase>,
cli_opts: &mut SimulatorCLI,
profile: &Profile,
) -> (u64, SimulatorEnv, InteractionPlan) {
if let Some(seed) = &cli_opts.load {
let seed = seed.parse::<u64>().expect("seed should be a number");
let bugbase = bugbase.expect("BugBase must be enabled to load a bug");
tracing::info!("seed={}", seed);
let bug = bugbase
.get_bug(seed)
.unwrap_or_else(|| panic!("bug '{seed}' not found in bug base"));
if let Some(seed) = cli_opts.load {
let bugbase = bugbase
.as_mut()
.expect("BugBase must be enabled to load a bug");
let paths = bugbase.paths(seed);
if !paths.base.exists() {
std::fs::create_dir_all(&paths.base).unwrap();
}
let env = SimulatorEnv::new(
bug.seed(),
cli_opts,
paths,
SimulationType::Default,
profile,
);
let plan = match bug {
Bug::Loaded(LoadedBug { plan, .. }) => plan.clone(),
Bug::Unloaded { seed } => {
let seed = *seed;
bugbase
.load_bug(seed)
.unwrap_or_else(|_| panic!("could not load bug '{seed}' in bug base"))
.plan
.clone()
let bug = bugbase
.get_or_load_bug(seed)
.unwrap()
.unwrap_or_else(|| panic!("bug '{seed}' not found in bug base"));
// run the simulation with the same CLI options as the loaded bug
*cli_opts = bug.last_cli_opts();
}
};
std::fs::write(env.get_plan_path(), plan.to_string()).unwrap();
std::fs::write(
env.get_plan_path().with_extension("json"),
serde_json::to_string_pretty(&plan).unwrap(),
)
.unwrap();
(seed, env, plan)
} else {
let seed = cli_opts.seed.unwrap_or_else(|| {
let mut rng = rand::rng();
rng.next_u64()
});
tracing::info!("seed={}", seed);
cli_opts.seed = Some(seed);
let paths = if let Some(bugbase) = bugbase {
let paths = bugbase.paths(seed);
@@ -535,14 +455,13 @@ fn setup_simulation(
Paths::new(&dir)
};
let mut env = SimulatorEnv::new(seed, cli_opts, paths, SimulationType::Default, profile);
let env = SimulatorEnv::new(seed, cli_opts, paths, SimulationType::Default, profile);
tracing::info!("Generating database interaction plan...");
let plan = InteractionPlan::init_plan(&mut env);
let plan = InteractionPlan::new(env.profile.experimental_mvcc);
(seed, env, plan)
}
}
fn run_simulation(

903
simulator/model/interactions.rs Normal file
View File

@@ -0,0 +1,903 @@
use std::{
fmt::{Debug, Display},
marker::PhantomData,
num::NonZeroUsize,
ops::{Deref, DerefMut, Range},
panic::RefUnwindSafe,
rc::Rc,
sync::Arc,
};
use indexmap::IndexSet;
use itertools::Itertools;
use serde::{Deserialize, Serialize};
use sql_generation::model::table::SimValue;
use turso_core::{Connection, Result, StepResult};
use crate::{
generation::Shadow,
model::{
Query, ResultSet,
metrics::InteractionStats,
property::{Property, PropertyDiscriminants},
},
runner::env::{ShadowTablesMut, SimConnection, SimulationType, SimulatorEnv},
};
#[derive(Debug, Clone)]
pub(crate) struct InteractionPlan {
plan: Vec<Interaction>,
stats: InteractionStats,
// In the future, this should probably be a stack of interactions
// so we can have nested properties
last_interactions: Option<Interactions>,
pub mvcc: bool,
/// Counts [Interactions]. Should not count transactions statements, just so we can generate more meaningful interactions per run
/// This field is only necessary and valid when generating interactions. For static iteration, we do not care about this field
len_properties: usize,
next_interaction_id: NonZeroUsize,
}
impl InteractionPlan {
pub(crate) fn new(mvcc: bool) -> Self {
Self {
plan: Vec::new(),
stats: InteractionStats::default(),
last_interactions: None,
mvcc,
len_properties: 0,
next_interaction_id: NonZeroUsize::new(1).unwrap(),
}
}
/// Count of interactions
#[inline]
pub fn len(&self) -> usize {
self.plan.len()
}
/// Count of properties
#[inline]
pub fn len_properties(&self) -> usize {
self.len_properties
}
pub fn next_property_id(&mut self) -> NonZeroUsize {
let id = self.next_interaction_id;
self.next_interaction_id = self
.next_interaction_id
.checked_add(1)
.expect("Generated too many interactions, that overflowed ID generation");
id
}
pub fn last_interactions(&self) -> Option<&Interactions> {
self.last_interactions.as_ref()
}
pub fn push_interactions(&mut self, interactions: Interactions) {
if !interactions.ignore() {
self.len_properties += 1;
}
self.last_interactions = Some(interactions);
}
pub fn push(&mut self, interaction: Interaction) {
self.plan.push(interaction);
}
/// Finds the range of interactions that are contained between the start and end spans for a given ID.
pub fn find_interactions_range(&self, id: NonZeroUsize) -> Range<usize> {
let interactions = self.interactions_list();
let idx = interactions
.binary_search_by_key(&id, |interaction| interaction.id())
.map_err(|_| format!("Interaction containing id `{id}` should be present"))
.unwrap();
let interaction = &interactions[idx];
let backward = || -> usize {
interactions
.iter()
.enumerate()
.rev()
.skip(interactions.len() - idx)
.find(|(_, interaction)| interaction.id() != id)
.map(|(idx, _)| idx.saturating_add(1))
.unwrap_or(idx)
};
let forward = || -> usize {
interactions
.iter()
.enumerate()
.skip(idx + 1)
.find(|(_, interaction)| interaction.id() != id)
.map(|(idx, _)| idx.saturating_sub(1))
.unwrap_or(idx)
};
let range = if interaction.property_meta.is_some() {
// go backward and find the interaction that is not the same id
let start_idx = backward();
// go forward and find the interaction that is not the same id
let end_idx = forward();
start_idx..end_idx + 1
} else {
idx..idx + 1
};
assert!(!range.is_empty());
range
}
/// Truncates up to a particular interaction
pub fn truncate(&mut self, len: usize) {
self.plan.truncate(len);
}
/// Used to remove a particular [Interactions]
pub fn remove_property(&mut self, id: NonZeroUsize) {
let range = self.find_interactions_range(id);
// Consume the drain iterator just to be sure
for _interaction in self.plan.drain(range) {}
}
pub fn retain_mut<F>(&mut self, f: F)
where
F: FnMut(&mut Interaction) -> bool,
{
self.plan.retain_mut(f);
}
#[inline]
pub fn interactions_list(&self) -> &[Interaction] {
&self.plan
}
pub fn iter_properties(
&self,
) -> IterProperty<
std::iter::Peekable<std::iter::Enumerate<std::slice::Iter<'_, Interaction>>>,
Forward,
> {
IterProperty {
iter: self.interactions_list().iter().enumerate().peekable(),
_direction: PhantomData,
}
}
pub fn rev_iter_properties(
&self,
) -> IterProperty<
std::iter::Peekable<
std::iter::Enumerate<std::iter::Rev<std::slice::Iter<'_, Interaction>>>,
>,
Backward,
> {
IterProperty {
iter: self.interactions_list().iter().rev().enumerate().peekable(),
_direction: PhantomData,
}
}
pub fn stats(&self) -> &InteractionStats {
&self.stats
}
pub fn stats_mut(&mut self) -> &mut InteractionStats {
&mut self.stats
}
pub fn static_iterator(&self) -> impl InteractionPlanIterator {
PlanIterator {
iter: self.interactions_list().to_vec().into_iter(),
}
}
}
pub struct Forward;
pub struct Backward;
pub struct IterProperty<I, Dir> {
iter: I,
_direction: PhantomData<Dir>,
}
impl<'a, I> IterProperty<I, Forward>
where
I: Iterator<Item = (usize, &'a Interaction)> + itertools::PeekingNext + std::fmt::Debug,
{
pub fn next_property(&mut self) -> Option<impl Iterator<Item = (usize, &'a Interaction)>> {
let (idx, interaction) = self.iter.next()?;
let id = interaction.id();
// get interactions with a particular property
let first = std::iter::once((idx, interaction));
let property_interactions = first.chain(
self.iter
.peeking_take_while(move |(_idx, interaction)| interaction.id() == id),
);
Some(property_interactions)
}
}
impl<'a, I> IterProperty<I, Backward>
where
I: Iterator<Item = (usize, &'a Interaction)>
+ DoubleEndedIterator
+ itertools::PeekingNext
+ std::fmt::Debug,
{
pub fn next_property(&mut self) -> Option<impl Iterator<Item = (usize, &'a Interaction)>> {
let (idx, interaction) = self.iter.next()?;
let id = interaction.id();
// get interactions with a particular id
let first = std::iter::once((idx, interaction));
let property_interactions = self
.iter
.peeking_take_while(move |(_idx, interaction)| interaction.id() == id)
.chain(first);
Some(property_interactions.into_iter())
}
}
pub trait InteractionPlanIterator {
fn next(&mut self, env: &mut SimulatorEnv) -> Option<Interaction>;
}
impl<T: InteractionPlanIterator> InteractionPlanIterator for &mut T {
#[inline]
fn next(&mut self, env: &mut SimulatorEnv) -> Option<Interaction> {
T::next(self, env)
}
}
pub struct PlanIterator<I: Iterator<Item = Interaction>> {
iter: I,
}
impl<I> InteractionPlanIterator for PlanIterator<I>
where
I: Iterator<Item = Interaction>,
{
#[inline]
fn next(&mut self, _env: &mut SimulatorEnv) -> Option<Interaction> {
self.iter.next()
}
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub struct InteractionPlanState {
pub interaction_pointer: usize,
}
#[derive(Debug, Default, Clone)]
pub struct ConnectionState {
pub stack: Vec<ResultSet>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Interactions {
pub connection_index: usize,
pub interactions: InteractionsType,
}
impl Interactions {
pub fn new(connection_index: usize, interactions: InteractionsType) -> Self {
Self {
connection_index,
interactions,
}
}
/// Whether the interaction needs to check the database tables
pub fn check_tables(&self) -> bool {
match &self.interactions {
InteractionsType::Property(property) => property.check_tables(),
InteractionsType::Query(..) | InteractionsType::Fault(..) => false,
}
}
/// Interactions that are not counted/ignored in the InteractionPlan.
/// Used in InteractionPlan to not count certain interactions to its length, as they are just auxiliary. This allows more
/// meaningful interactions to be generation
fn ignore(&self) -> bool {
self.is_transaction()
|| matches!(
self.interactions,
InteractionsType::Property(Property::AllTableHaveExpectedContent { .. })
)
}
}
impl Deref for Interactions {
type Target = InteractionsType;
fn deref(&self) -> &Self::Target {
&self.interactions
}
}
impl DerefMut for Interactions {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.interactions
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum InteractionsType {
Property(Property),
Query(Query),
Fault(Fault),
}
impl InteractionsType {
pub fn is_transaction(&self) -> bool {
match self {
InteractionsType::Query(query) => query.is_transaction(),
_ => false,
}
}
}
impl Display for InteractionPlan {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
const PAD: usize = 4;
let mut indentation_level: usize = 0;
let mut iter = self.iter_properties();
while let Some(property) = iter.next_property() {
let mut property = property.peekable();
let mut start = true;
while let Some((_, interaction)) = property.next() {
if let Some(name) = interaction.property_meta.map(|p| p.property.name())
&& start
{
indentation_level = indentation_level.saturating_add(1);
writeln!(f, "-- begin testing '{name}'")?;
start = false;
}
if indentation_level > 0 {
let padding = " ".repeat(indentation_level * PAD);
f.pad(&padding)?;
}
writeln!(f, "{interaction}")?;
if let Some(name) = interaction.property_meta.map(|p| p.property.name())
&& property.peek().is_none()
{
indentation_level = indentation_level.saturating_sub(1);
writeln!(f, "-- end testing '{name}'")?;
}
}
}
Ok(())
}
}
type AssertionFunc =
dyn Fn(&Vec<ResultSet>, &mut SimulatorEnv) -> Result<Result<(), String>> + RefUnwindSafe;
#[derive(Clone)]
pub struct Assertion {
pub func: Rc<AssertionFunc>,
pub name: String, // For display purposes in the plan
pub tables: Vec<String>, // Tables it depends on
}
impl Debug for Assertion {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Assertion")
.field("name", &self.name)
.finish()
}
}
impl Assertion {
pub fn new<F>(name: String, func: F, tables: Vec<String>) -> Self
where
F: Fn(&Vec<ResultSet>, &mut SimulatorEnv) -> Result<Result<(), String>>
+ 'static
+ RefUnwindSafe,
{
Self {
func: Rc::new(func),
name,
tables,
}
}
pub fn dependencies(&self) -> IndexSet<String> {
IndexSet::from_iter(self.tables.clone())
}
pub fn uses(&self) -> Vec<String> {
self.tables.clone()
}
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
pub enum Fault {
Disconnect,
ReopenDatabase,
}
impl Display for Fault {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Fault::Disconnect => write!(f, "DISCONNECT"),
Fault::ReopenDatabase => write!(f, "REOPEN_DATABASE"),
}
}
}
#[derive(Debug, Clone, Copy)]
pub struct PropertyMetadata {
pub property: PropertyDiscriminants,
// If the query is an extension query
pub extension: bool,
}
impl PropertyMetadata {
pub fn new(property: &Property, extension: bool) -> PropertyMetadata {
Self {
property: property.into(),
extension,
}
}
}
#[derive(Debug, Clone, derive_builder::Builder)]
pub struct Interaction {
pub connection_index: usize,
pub interaction: InteractionType,
#[builder(default)]
pub ignore_error: bool,
#[builder(setter(strip_option), default)]
pub property_meta: Option<PropertyMetadata>,
/// 0 id means the ID was not set
id: NonZeroUsize,
}
impl InteractionBuilder {
pub fn from_interaction(interaction: &Interaction) -> Self {
let mut builder = Self::default();
builder
.connection_index(interaction.connection_index)
.id(interaction.id())
.ignore_error(interaction.ignore_error)
.interaction(interaction.interaction.clone());
if let Some(property_meta) = interaction.property_meta {
builder.property_meta(property_meta);
}
builder
}
pub fn with_interaction(interaction: InteractionType) -> Self {
let mut builder = Self::default();
builder.interaction(interaction);
builder
}
/// Checks to see if the property metadata was already set
pub fn has_property_meta(&self) -> bool {
self.property_meta.is_some()
}
}
impl Deref for Interaction {
type Target = InteractionType;
fn deref(&self) -> &Self::Target {
&self.interaction
}
}
impl DerefMut for Interaction {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.interaction
}
}
impl Interaction {
pub fn id(&self) -> NonZeroUsize {
self.id
}
pub fn uses(&self) -> Vec<String> {
match &self.interaction {
InteractionType::Query(query)
| InteractionType::FsyncQuery(query)
| InteractionType::FaultyQuery(query) => query.uses(),
InteractionType::Assertion(assert) | InteractionType::Assumption(assert) => {
assert.uses()
}
_ => vec![],
}
}
}
#[derive(Debug, Clone)]
pub enum InteractionType {
Query(Query),
Assumption(Assertion),
Assertion(Assertion),
Fault(Fault),
/// Will attempt to run any random query. However, when the connection tries to sync it will
/// close all connections and reopen the database and assert that no data was lost
FsyncQuery(Query),
FaultyQuery(Query),
}
// FIXME: add the connection index here later
impl Display for Interaction {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}; -- {}", self.interaction, self.connection_index)
}
}
impl Display for InteractionType {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Query(query) => write!(f, "{query}"),
Self::Assumption(assumption) => write!(f, "-- ASSUME {}", assumption.name),
Self::Assertion(assertion) => {
write!(f, "-- ASSERT {};", assertion.name)
}
Self::Fault(fault) => write!(f, "-- FAULT '{fault}'"),
Self::FsyncQuery(query) => {
writeln!(f, "-- FSYNC QUERY")?;
writeln!(f, "{query};")?;
write!(f, "{query};")
}
Self::FaultyQuery(query) => write!(f, "{query}; -- FAULTY QUERY"),
}
}
}
impl Shadow for InteractionType {
type Result = anyhow::Result<Vec<Vec<SimValue>>>;
fn shadow(&self, env: &mut ShadowTablesMut) -> Self::Result {
match self {
Self::Query(query) => {
if !query.is_transaction() {
env.add_query(query);
}
query.shadow(env)
}
Self::Assumption(_)
| Self::Assertion(_)
| Self::Fault(_)
| Self::FaultyQuery(_)
| Self::FsyncQuery(_) => Ok(vec![]),
}
}
}
impl InteractionType {
pub fn is_ddl(&self) -> bool {
match self {
InteractionType::Query(query)
| InteractionType::FsyncQuery(query)
| InteractionType::FaultyQuery(query) => query.is_ddl(),
_ => false,
}
}
pub(crate) fn execute_query(&self, conn: &mut Arc<Connection>) -> ResultSet {
if let Self::Query(query) = self {
assert!(
!matches!(query, Query::Placeholder),
"simulation cannot have a placeholder Query for execution"
);
let query_str = query.to_string();
let rows = conn.query(&query_str);
if rows.is_err() {
let err = rows.err();
tracing::debug!(
"Error running query '{}': {:?}",
&query_str[0..query_str.len().min(4096)],
err
);
// Do not panic on parse error, because DoubleCreateFailure relies on it
return Err(err.unwrap());
}
let rows = rows?;
assert!(rows.is_some());
let mut rows = rows.unwrap();
let mut out = Vec::new();
while let Ok(row) = rows.step() {
match row {
StepResult::Row => {
let row = rows.row().unwrap();
let mut r = Vec::new();
for v in row.get_values() {
let v = v.into();
r.push(v);
}
out.push(r);
}
StepResult::IO => {
rows.run_once().unwrap();
}
StepResult::Interrupt => {}
StepResult::Done => {
break;
}
StepResult::Busy => {
return Err(turso_core::LimboError::Busy);
}
}
}
Ok(out)
} else {
unreachable!("unexpected: this function should only be called on queries")
}
}
pub(crate) fn execute_assertion(
&self,
stack: &Vec<ResultSet>,
env: &mut SimulatorEnv,
) -> Result<()> {
match self {
Self::Assertion(assertion) => {
let result = assertion.func.as_ref()(stack, env);
match result {
Ok(Ok(())) => Ok(()),
Ok(Err(message)) => Err(turso_core::LimboError::InternalError(format!(
"Assertion '{}' failed: {}",
assertion.name, message
))),
Err(err) => Err(turso_core::LimboError::InternalError(format!(
"Assertion '{}' execution error: {}",
assertion.name, err
))),
}
}
_ => {
unreachable!("unexpected: this function should only be called on assertions")
}
}
}
pub(crate) fn execute_assumption(
&self,
stack: &Vec<ResultSet>,
env: &mut SimulatorEnv,
) -> Result<()> {
match self {
Self::Assumption(assumption) => {
let result = assumption.func.as_ref()(stack, env);
match result {
Ok(Ok(())) => Ok(()),
Ok(Err(message)) => Err(turso_core::LimboError::InternalError(format!(
"Assumption '{}' failed: {}",
assumption.name, message
))),
Err(err) => Err(turso_core::LimboError::InternalError(format!(
"Assumption '{}' execution error: {}",
assumption.name, err
))),
}
}
_ => {
unreachable!("unexpected: this function should only be called on assumptions")
}
}
}
pub(crate) fn execute_fault(&self, env: &mut SimulatorEnv, conn_index: usize) -> Result<()> {
match self {
Self::Fault(fault) => {
match fault {
Fault::Disconnect => {
if env.connections[conn_index].is_connected() {
if env.conn_in_transaction(conn_index) {
env.rollback_conn(conn_index);
}
env.connections[conn_index].disconnect();
} else {
return Err(turso_core::LimboError::InternalError(
"connection already disconnected".into(),
));
}
}
Fault::ReopenDatabase => {
reopen_database(env);
}
}
Ok(())
}
_ => {
unreachable!("unexpected: this function should only be called on faults")
}
}
}
pub(crate) fn execute_fsync_query(
&self,
conn: Arc<Connection>,
env: &mut SimulatorEnv,
) -> ResultSet {
if let Self::FsyncQuery(query) = self {
let query_str = query.to_string();
let rows = conn.query(&query_str);
if rows.is_err() {
let err = rows.err();
tracing::debug!(
"Error running query '{}': {:?}",
&query_str[0..query_str.len().min(4096)],
err
);
return Err(err.unwrap());
}
let mut rows = rows.unwrap().unwrap();
let mut out = Vec::new();
while let Ok(row) = rows.step() {
match row {
StepResult::Row => {
let row = rows.row().unwrap();
let mut r = Vec::new();
for v in row.get_values() {
let v = v.into();
r.push(v);
}
out.push(r);
}
StepResult::IO => {
let syncing = env.io.syncing();
if syncing {
reopen_database(env);
} else {
rows.run_once().unwrap();
}
}
StepResult::Done => {
break;
}
StepResult::Busy => {
return Err(turso_core::LimboError::Busy);
}
StepResult::Interrupt => {}
}
}
Ok(out)
} else {
unreachable!("unexpected: this function should only be called on queries")
}
}
pub(crate) fn execute_faulty_query(
&self,
conn: &Arc<Connection>,
env: &mut SimulatorEnv,
) -> ResultSet {
use rand::Rng;
if let Self::FaultyQuery(query) = self {
let query_str = query.to_string();
let rows = conn.query(&query_str);
if rows.is_err() {
let err = rows.err();
tracing::debug!(
"Error running query '{}': {:?}",
&query_str[0..query_str.len().min(4096)],
err
);
if let Some(turso_core::LimboError::ParseError(e)) = err {
panic!("Unexpected parse error: {e}");
}
return Err(err.unwrap());
}
let mut rows = rows.unwrap().unwrap();
let mut out = Vec::new();
let mut current_prob = 0.05;
let mut incr = 0.001;
loop {
let syncing = env.io.syncing();
let inject_fault = env.rng.random_bool(current_prob);
// TODO: avoid for now injecting faults when syncing
if inject_fault && !syncing {
env.io.inject_fault(true);
}
match rows.step()? {
StepResult::Row => {
let row = rows.row().unwrap();
let mut r = Vec::new();
for v in row.get_values() {
let v = v.into();
r.push(v);
}
out.push(r);
}
StepResult::IO => {
rows.run_once()?;
current_prob += incr;
if current_prob > 1.0 {
current_prob = 1.0;
} else {
incr *= 1.01;
}
}
StepResult::Done => {
break;
}
StepResult::Busy => {
return Err(turso_core::LimboError::Busy);
}
StepResult::Interrupt => {}
}
}
Ok(out)
} else {
unreachable!("unexpected: this function should only be called on queries")
}
}
}
fn reopen_database(env: &mut SimulatorEnv) {
// 1. Close all connections without default checkpoint-on-close behavior
// to expose bugs related to how we handle WAL
let mvcc = env.profile.experimental_mvcc;
let indexes = env.profile.query.gen_opts.indexes;
let num_conns = env.connections.len();
env.connections.clear();
// Clear all open files
// TODO: for correct reporting of faults we should get all the recorded numbers and transfer to the new file
env.io.close_files();
// 2. Re-open database
match env.type_ {
SimulationType::Differential => {
for _ in 0..num_conns {
env.connections.push(SimConnection::SQLiteConnection(
rusqlite::Connection::open(env.get_db_path())
.expect("Failed to open SQLite connection"),
));
}
}
SimulationType::Default | SimulationType::Doublecheck => {
env.db = None;
let db = match turso_core::Database::open_file_with_flags(
env.io.clone(),
env.get_db_path().to_str().expect("path should be 'to_str'"),
turso_core::OpenFlags::default(),
turso_core::DatabaseOpts::new()
.with_mvcc(mvcc)
.with_indexes(indexes)
.with_autovacuum(true),
None,
) {
Ok(db) => db,
Err(e) => {
tracing::error!(
"Failed to open database at {}: {}",
env.get_db_path().display(),
e
);
panic!("Failed to open database: {e}");
}
};
env.db = Some(db);
for _ in 0..num_conns {
env.connections.push(SimConnection::LimboConnection(
env.db.as_ref().expect("db to be Some").connect().unwrap(),
));
}
}
};
}

175
simulator/model/metrics.rs Normal file
View File

@@ -0,0 +1,175 @@
use std::fmt::Display;
use sql_generation::generation::GenerationContext;
use crate::{
model::{
Query,
interactions::{Interaction, InteractionType},
},
profiles::query::QueryProfile,
};
#[derive(Debug)]
pub struct Remaining {
pub select: u32,
pub insert: u32,
pub create: u32,
pub create_index: u32,
pub delete: u32,
pub update: u32,
pub drop: u32,
pub alter_table: u32,
pub drop_index: u32,
pub pragma_count: u32,
}
impl Remaining {
pub fn new(
max_interactions: u32,
opts: &QueryProfile,
stats: &InteractionStats,
mvcc: bool,
context: &impl GenerationContext,
) -> Remaining {
let total_weight = opts.total_weight();
let total_select = (max_interactions * opts.select_weight) / total_weight;
let total_insert = (max_interactions * opts.insert_weight) / total_weight;
let total_create = (max_interactions * opts.create_table_weight) / total_weight;
let total_create_index = (max_interactions * opts.create_index_weight) / total_weight;
let total_delete = (max_interactions * opts.delete_weight) / total_weight;
let total_update = (max_interactions * opts.update_weight) / total_weight;
let total_drop = (max_interactions * opts.drop_table_weight) / total_weight;
let total_alter_table = (max_interactions * opts.alter_table_weight) / total_weight;
let total_drop_index = (max_interactions * opts.drop_index) / total_weight;
let total_pragma = (max_interactions * opts.pragma_weight) / total_weight;
let remaining_select = total_select
.checked_sub(stats.select_count)
.unwrap_or_default();
let remaining_insert = total_insert
.checked_sub(stats.insert_count)
.unwrap_or_default();
let remaining_create = total_create
.checked_sub(stats.create_count)
.unwrap_or_default();
let mut remaining_create_index = total_create_index
.checked_sub(stats.create_index_count)
.unwrap_or_default();
let remaining_delete = total_delete
.checked_sub(stats.delete_count)
.unwrap_or_default();
let remaining_update = total_update
.checked_sub(stats.update_count)
.unwrap_or_default();
let remaining_drop = total_drop.checked_sub(stats.drop_count).unwrap_or_default();
let remaining_pragma = total_pragma
.checked_sub(stats.pragma_count)
.unwrap_or_default();
let remaining_alter_table = total_alter_table
.checked_sub(stats.alter_table_count)
.unwrap_or_default();
let mut remaining_drop_index = total_drop_index
.checked_sub(stats.alter_table_count)
.unwrap_or_default();
if mvcc {
// TODO: index not supported yet for mvcc
remaining_create_index = 0;
remaining_drop_index = 0;
}
// if there are no indexes do not allow creation of drop_index
if !context
.tables()
.iter()
.any(|table| !table.indexes.is_empty())
{
remaining_drop_index = 0;
}
Remaining {
select: remaining_select,
insert: remaining_insert,
create: remaining_create,
create_index: remaining_create_index,
delete: remaining_delete,
drop: remaining_drop,
update: remaining_update,
alter_table: remaining_alter_table,
drop_index: remaining_drop_index,
pragma_count: remaining_pragma,
}
}
}
#[derive(Debug, Clone, Copy, Default)]
pub(crate) struct InteractionStats {
pub select_count: u32,
pub insert_count: u32,
pub delete_count: u32,
pub update_count: u32,
pub create_count: u32,
pub create_index_count: u32,
pub drop_count: u32,
pub begin_count: u32,
pub commit_count: u32,
pub rollback_count: u32,
pub alter_table_count: u32,
pub drop_index_count: u32,
pub pragma_count: u32,
}
impl InteractionStats {
pub fn update(&mut self, interaction: &Interaction) {
match &interaction.interaction {
InteractionType::Query(query)
| InteractionType::FsyncQuery(query)
| InteractionType::FaultyQuery(query) => self.query_stat(query),
_ => {}
}
}
fn query_stat(&mut self, q: &Query) {
match q {
Query::Select(_) => self.select_count += 1,
Query::Insert(_) => self.insert_count += 1,
Query::Delete(_) => self.delete_count += 1,
Query::Create(_) => self.create_count += 1,
Query::Drop(_) => self.drop_count += 1,
Query::Update(_) => self.update_count += 1,
Query::CreateIndex(_) => self.create_index_count += 1,
Query::Begin(_) => self.begin_count += 1,
Query::Commit(_) => self.commit_count += 1,
Query::Rollback(_) => self.rollback_count += 1,
Query::AlterTable(_) => self.alter_table_count += 1,
Query::DropIndex(_) => self.drop_index_count += 1,
Query::Placeholder => {}
Query::Pragma(_) => self.pragma_count += 1,
}
}
}
impl Display for InteractionStats {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"Read: {}, Insert: {}, Delete: {}, Update: {}, Create: {}, CreateIndex: {}, Drop: {}, Begin: {}, Commit: {}, Rollback: {}, Alter Table: {}, Drop Index: {}",
self.select_count,
self.insert_count,
self.delete_count,
self.update_count,
self.create_count,
self.create_index_count,
self.drop_count,
self.begin_count,
self.commit_count,
self.rollback_count,
self.alter_table_count,
self.drop_index_count,
)
}
}

6
simulator/model/mod.rs
View File

@@ -20,6 +20,12 @@ use turso_parser::ast::Distinctness;
use crate::{generation::Shadow, runner::env::ShadowTablesMut};
pub mod interactions;
pub mod metrics;
pub mod property;
pub(crate) type ResultSet = turso_core::Result<Vec<Vec<SimValue>>>;
// This type represents the potential queries on the database.
#[derive(Debug, Clone, Serialize, Deserialize, strum::EnumDiscriminants)]
pub enum Query {

245
simulator/model/property.rs Normal file
View File

@@ -0,0 +1,245 @@
use serde::{Deserialize, Serialize};
use sql_generation::model::query::{Create, Insert, Select, predicate::Predicate, update::Update};
use crate::model::Query;
/// Properties are representations of executable specifications
/// about the database behavior.
#[derive(Debug, Clone, Serialize, Deserialize, strum::EnumDiscriminants, strum::IntoStaticStr)]
#[strum_discriminants(derive(strum::EnumIter, strum::IntoStaticStr))]
#[strum(serialize_all = "Train-Case")]
pub enum Property {
/// Insert-Select is a property in which the inserted row
/// must be in the resulting rows of a select query that has a
/// where clause that matches the inserted row.
/// The execution of the property is as follows
/// INSERT INTO <t> VALUES (...)
/// I_0
/// I_1
/// ...
/// I_n
/// SELECT * FROM <t> WHERE <predicate>
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - The inserted row will not be deleted.
/// - The inserted row will not be updated.
/// - The table `t` will not be renamed, dropped, or altered.
InsertValuesSelect {
/// The insert query
insert: Insert,
/// Selected row index
row_index: usize,
/// Additional interactions in the middle of the property
queries: Vec<Query>,
/// The select query
select: Select,
/// Interactive query information if any
interactive: Option<InteractiveQueryInfo>,
},
/// ReadYourUpdatesBack is a property in which the updated rows
/// must be in the resulting rows of a select query that has a
/// where clause that matches the updated row.
/// The execution of the property is as follows
/// UPDATE <t> SET <set_cols=set_vals> WHERE <predicate>
/// SELECT <set_cols> FROM <t> WHERE <predicate>
/// These interactions are executed in immediate succession
/// just to verify the property that our updates did what they
/// were supposed to do.
ReadYourUpdatesBack {
update: Update,
select: Select,
},
/// TableHasExpectedContent is a property in which the table
/// must have the expected content, i.e. all the insertions and
/// updates and deletions should have been persisted in the way
/// we think they were.
/// The execution of the property is as follows
/// SELECT * FROM <t>
/// ASSERT <expected_content>
TableHasExpectedContent {
table: String,
},
/// AllTablesHaveExpectedContent is a property in which the table
/// must have the expected content, i.e. all the insertions and
/// updates and deletions should have been persisted in the way
/// we think they were.
/// The execution of the property is as follows
/// SELECT * FROM <t>
/// ASSERT <expected_content>
/// for each table in the simulator model
AllTableHaveExpectedContent {
tables: Vec<String>,
},
/// Double Create Failure is a property in which creating
/// the same table twice leads to an error.
/// The execution of the property is as follows
/// CREATE TABLE <t> (...)
/// I_0
/// I_1
/// ...
/// I_n
/// CREATE TABLE <t> (...) -> Error
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - Table `t` will not be renamed or dropped.
DoubleCreateFailure {
/// The create query
create: Create,
/// Additional interactions in the middle of the property
queries: Vec<Query>,
},
/// Select Limit is a property in which the select query
/// has a limit clause that is respected by the query.
/// The execution of the property is as follows
/// SELECT * FROM <t> WHERE <predicate> LIMIT <n>
/// This property is a single-interaction property.
/// The interaction has the following constraints;
/// - The select query will respect the limit clause.
SelectLimit {
/// The select query
select: Select,
},
/// Delete-Select is a property in which the deleted row
/// must not be in the resulting rows of a select query that has a
/// where clause that matches the deleted row. In practice, `p1` of
/// the delete query will be used as the predicate for the select query,
/// hence the select should return NO ROWS.
/// The execution of the property is as follows
/// DELETE FROM <t> WHERE <predicate>
/// I_0
/// I_1
/// ...
/// I_n
/// SELECT * FROM <t> WHERE <predicate>
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - A row that holds for the predicate will not be inserted.
/// - The table `t` will not be renamed, dropped, or altered.
DeleteSelect {
table: String,
predicate: Predicate,
queries: Vec<Query>,
},
/// Drop-Select is a property in which selecting from a dropped table
/// should result in an error.
/// The execution of the property is as follows
/// DROP TABLE <t>
/// I_0
/// I_1
/// ...
/// I_n
/// SELECT * FROM <t> WHERE <predicate> -> Error
/// The interactions in the middle has the following constraints;
/// - There will be no errors in the middle interactions.
/// - The table `t` will not be created, no table will be renamed to `t`.
DropSelect {
table: String,
queries: Vec<Query>,
select: Select,
},
/// Select-Select-Optimizer is a property in which we test the optimizer by
/// running two equivalent select queries, one with `SELECT <predicate> from <t>`
/// and the other with `SELECT * from <t> WHERE <predicate>`. As highlighted by
/// Rigger et al. in Non-Optimizing Reference Engine Construction(NoREC), SQLite
/// tends to optimize `where` statements while keeping the result column expressions
/// unoptimized. This property is used to test the optimizer. The property is successful
/// if the two queries return the same number of rows.
SelectSelectOptimizer {
table: String,
predicate: Predicate,
},
/// Where-True-False-Null is a property that tests the boolean logic implementation
/// in the database. It relies on the fact that `P == true || P == false || P == null` should return true,
/// as SQLite uses a ternary logic system. This property is invented in "Finding Bugs in Database Systems via Query Partitioning"
/// by Rigger et al. and it is canonically called Ternary Logic Partitioning (TLP).
WhereTrueFalseNull {
select: Select,
predicate: Predicate,
},
/// UNION-ALL-Preserves-Cardinality is a property that tests the UNION ALL operator
/// implementation in the database. It relies on the fact that `SELECT * FROM <t
/// > WHERE <predicate> UNION ALL SELECT * FROM <t> WHERE <predicate>`
/// should return the same number of rows as `SELECT <predicate> FROM <t> WHERE <predicate>`.
/// > The property is succesfull when the UNION ALL of 2 select queries returns the same number of rows
/// > as the sum of the two select queries.
UnionAllPreservesCardinality {
select: Select,
where_clause: Predicate,
},
/// FsyncNoWait is a property which tests if we do not loose any data after not waiting for fsync.
///
/// # Interactions
/// - Executes the `query` without waiting for fsync
/// - Drop all connections and Reopen the database
/// - Execute the `query` again
/// - Query tables to assert that the values were inserted
///
FsyncNoWait {
query: Query,
},
FaultyQuery {
query: Query,
},
/// Property used to subsititute a property with its queries only
Queries {
queries: Vec<Query>,
},
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct InteractiveQueryInfo {
pub start_with_immediate: bool,
pub end_with_commit: bool,
}
impl Property {
/// Property Does some sort of fault injection
pub fn check_tables(&self) -> bool {
matches!(
self,
Property::FsyncNoWait { .. } | Property::FaultyQuery { .. }
)
}
pub fn has_extensional_queries(&self) -> bool {
matches!(
self,
Property::InsertValuesSelect { .. }
| Property::DoubleCreateFailure { .. }
| Property::DeleteSelect { .. }
| Property::DropSelect { .. }
| Property::Queries { .. }
)
}
pub fn get_extensional_queries(&mut self) -> Option<&mut Vec<Query>> {
match self {
Property::InsertValuesSelect { queries, .. }
| Property::DoubleCreateFailure { queries, .. }
| Property::DeleteSelect { queries, .. }
| Property::DropSelect { queries, .. }
| Property::Queries { queries } => Some(queries),
Property::FsyncNoWait { .. } | Property::FaultyQuery { .. } => None,
Property::SelectLimit { .. }
| Property::SelectSelectOptimizer { .. }
| Property::WhereTrueFalseNull { .. }
| Property::UnionAllPreservesCardinality { .. }
| Property::ReadYourUpdatesBack { .. }
| Property::TableHasExpectedContent { .. }
| Property::AllTableHaveExpectedContent { .. } => None,
}
}
}
impl PropertyDiscriminants {
pub fn name(&self) -> &'static str {
self.into()
}
pub fn check_tables(&self) -> bool {
matches!(
self,
Self::AllTableHaveExpectedContent | Self::TableHasExpectedContent
)
}
}

365
simulator/runner/bugbase.rs
View File

@@ -2,7 +2,7 @@ use std::{
collections::HashMap,
env::current_dir,
fs::File,
io::{self, Read, Write},
io::Read,
path::{Path, PathBuf},
time::SystemTime,
};
@@ -11,60 +11,84 @@ use anyhow::{Context, anyhow};
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use crate::{InteractionPlan, Paths};
use crate::{Paths, model::interactions::InteractionPlan};
use super::cli::SimulatorCLI;
const READABLE_PLAN_PATH: &str = "plan.sql";
const SHRUNK_READABLE_PLAN_PATH: &str = "shrunk.sql";
const SEED_PATH: &str = "seed.txt";
const RUNS_PATH: &str = "runs.json";
/// A bug is a run that has been identified as buggy.
#[derive(Clone)]
pub(crate) enum Bug {
Unloaded { seed: u64 },
Loaded(LoadedBug),
}
#[derive(Clone)]
pub struct LoadedBug {
pub struct Bug {
/// The seed of the bug.
pub seed: u64,
/// The plan of the bug.
pub plan: InteractionPlan,
/// TODO: currently plan is only saved to the .sql file, and that is not deserializable yet
/// so we cannot always store an interaction plan here
pub plan: Option<InteractionPlan>,
/// The shrunk plan of the bug, if any.
pub shrunk_plan: Option<InteractionPlan>,
/// The runs of the bug.
pub runs: Vec<BugRun>,
}
#[derive(Clone, Serialize, Deserialize)]
pub(crate) struct BugRun {
pub struct BugRun {
/// Commit hash of the current version of Limbo.
pub(crate) hash: String,
pub hash: String,
/// Timestamp of the run.
#[serde(with = "chrono::serde::ts_seconds")]
pub(crate) timestamp: DateTime<Utc>,
pub timestamp: DateTime<Utc>,
/// Error message of the run.
pub(crate) error: Option<String>,
pub error: Option<String>,
/// Options
pub(crate) cli_options: SimulatorCLI,
pub cli_options: SimulatorCLI,
/// Whether the run was a shrunk run.
pub(crate) shrunk: bool,
pub shrunk: bool,
}
impl Bug {
#[expect(dead_code)]
/// Check if the bug is loaded.
pub(crate) fn is_loaded(&self) -> bool {
match self {
Bug::Unloaded { .. } => false,
Bug::Loaded { .. } => true,
}
fn save_to_path(&self, path: impl AsRef<Path>) -> anyhow::Result<()> {
let path = path.as_ref();
let bug_path = path.join(self.seed.to_string());
std::fs::create_dir_all(&bug_path)
.with_context(|| "should be able to create bug directory")?;
let seed_path = bug_path.join(SEED_PATH);
std::fs::write(&seed_path, self.seed.to_string())
.with_context(|| "should be able to write seed file")?;
if let Some(plan) = &self.plan {
let readable_plan_path = bug_path.join(READABLE_PLAN_PATH);
std::fs::write(&readable_plan_path, plan.to_string())
.with_context(|| "should be able to write readable plan file")?;
}
/// Get the seed of the bug.
pub(crate) fn seed(&self) -> u64 {
match self {
Bug::Unloaded { seed } => *seed,
Bug::Loaded(LoadedBug { seed, .. }) => *seed,
if let Some(shrunk_plan) = &self.shrunk_plan {
let readable_shrunk_plan_path = bug_path.join(SHRUNK_READABLE_PLAN_PATH);
std::fs::write(&readable_shrunk_plan_path, shrunk_plan.to_string())
.with_context(|| "should be able to write readable shrunk plan file")?;
}
let runs_path = bug_path.join(RUNS_PATH);
std::fs::write(
&runs_path,
serde_json::to_string_pretty(&self.runs)
.with_context(|| "should be able to serialize runs")?,
)
.with_context(|| "should be able to write runs file")?;
Ok(())
}
pub fn last_cli_opts(&self) -> SimulatorCLI {
self.runs.last().unwrap().cli_options.clone()
}
}
@@ -73,13 +97,14 @@ pub(crate) struct BugBase {
/// Path to the bug base directory.
path: PathBuf,
/// The list of buggy runs, uniquely identified by their seed
bugs: HashMap<u64, Bug>,
bugs: HashMap<u64, Option<Bug>>,
}
impl BugBase {
/// Create a new bug base.
fn new(path: PathBuf) -> anyhow::Result<Self> {
let mut bugs = HashMap::new();
// list all the bugs in the path as directories
if let Ok(entries) = std::fs::read_dir(&path) {
for entry in entries.flatten() {
@@ -95,7 +120,7 @@ impl BugBase {
entry.file_name().to_string_lossy()
)
})?;
bugs.insert(seed, Bug::Unloaded { seed });
bugs.insert(seed, None);
}
}
}
@@ -105,7 +130,7 @@ impl BugBase {
/// Load the bug base from one of the potential paths.
pub(crate) fn load() -> anyhow::Result<Self> {
let potential_paths = vec![
let potential_paths = [
// limbo project directory
BugBase::get_limbo_project_dir()?,
// home directory
@@ -132,57 +157,33 @@ impl BugBase {
Err(anyhow!("failed to create bug base"))
}
#[expect(dead_code)]
/// Load the bug base from one of the potential paths.
pub(crate) fn interactive_load() -> anyhow::Result<Self> {
let potential_paths = vec![
// limbo project directory
BugBase::get_limbo_project_dir()?,
// home directory
dirs::home_dir().with_context(|| "should be able to get home directory")?,
// current directory
std::env::current_dir().with_context(|| "should be able to get current directory")?,
];
fn load_bug(&self, seed: u64) -> anyhow::Result<Bug> {
let path = self.path.join(seed.to_string()).join(RUNS_PATH);
for path in potential_paths {
let path = path.join(".bugbase");
if path.exists() {
return BugBase::new(path);
}
}
println!("select bug base location:");
println!("1. limbo project directory");
println!("2. home directory");
println!("3. current directory");
print!("> ");
io::stdout().flush().unwrap();
let mut choice = String::new();
io::stdin()
.read_line(&mut choice)
.expect("failed to read line");
let choice = choice
.trim()
.parse::<u32>()
.with_context(|| format!("invalid choice {choice}"))?;
let path = match choice {
1 => BugBase::get_limbo_project_dir()?.join(".bugbase"),
2 => {
let home = std::env::var("HOME").with_context(|| "failed to get home directory")?;
PathBuf::from(home).join(".bugbase")
}
3 => PathBuf::from(".bugbase"),
_ => anyhow::bail!(format!("invalid choice {choice}")),
let runs = if !path.exists() {
vec![]
} else {
std::fs::read_to_string(self.path.join(seed.to_string()).join(RUNS_PATH))
.with_context(|| "should be able to read runs file")
.and_then(|runs| serde_json::from_str(&runs).map_err(|e| anyhow!("{}", e)))?
};
if path.exists() {
unreachable!("bug base already exists at {}", path.display());
} else {
std::fs::create_dir_all(&path).with_context(|| "failed to create bug base")?;
tracing::info!("bug base created at {}", path.display());
BugBase::new(path)
let bug = Bug {
seed,
plan: None,
shrunk_plan: None,
runs,
};
Ok(bug)
}
pub fn load_bugs(&self) -> anyhow::Result<Vec<Bug>> {
let seeds = self.bugs.keys().copied().collect::<Vec<_>>();
seeds
.iter()
.map(|seed| self.load_bug(*seed))
.collect::<Result<Vec<_>, _>>()
}
/// Add a new bug to the bug base.
@@ -193,12 +194,11 @@ impl BugBase {
error: Option<String>,
cli_options: &SimulatorCLI,
) -> anyhow::Result<()> {
tracing::debug!("adding bug with seed {}", seed);
let bug = self.get_bug(seed);
let path = self.path.clone();
if bug.is_some() {
let mut bug = self.load_bug(seed)?;
bug.plan = plan.clone();
tracing::debug!("adding bug with seed {}", seed);
let bug = self.get_or_load_bug(seed)?;
let bug = if let Some(bug) = bug {
bug.runs.push(BugRun {
hash: Self::get_current_commit_hash()?,
timestamp: SystemTime::now().into(),
@@ -206,11 +206,13 @@ impl BugBase {
cli_options: cli_options.clone(),
shrunk: false,
});
self.bugs.insert(seed, Bug::Loaded(bug.clone()));
bug.plan = Some(plan);
bug
} else {
let bug = LoadedBug {
let bug = Bug {
seed,
plan: plan.clone(),
plan: Some(plan),
shrunk_plan: None,
runs: vec![BugRun {
hash: Self::get_current_commit_hash()?,
timestamp: SystemTime::now().into(),
@@ -218,172 +220,44 @@ impl BugBase {
cli_options: cli_options.clone(),
shrunk: false,
}],
shrunk_plan: None,
};
self.bugs.insert(seed, Bug::Loaded(bug.clone()));
}
// Save the bug to the bug base.
self.save_bug(seed)
}
/// Get a bug from the bug base.
pub(crate) fn get_bug(&self, seed: u64) -> Option<&Bug> {
self.bugs.get(&seed)
}
/// Save a bug to the bug base.
fn save_bug(&self, seed: u64) -> anyhow::Result<()> {
let bug = self.get_bug(seed);
match bug {
None | Some(Bug::Unloaded { .. }) => {
unreachable!("save should only be called within add_bug");
}
Some(Bug::Loaded(bug)) => {
let bug_path = self.path.join(seed.to_string());
std::fs::create_dir_all(&bug_path)
.with_context(|| "should be able to create bug directory")?;
let seed_path = bug_path.join("seed.txt");
std::fs::write(&seed_path, seed.to_string())
.with_context(|| "should be able to write seed file")?;
let plan_path = bug_path.join("plan.json");
std::fs::write(
&plan_path,
serde_json::to_string_pretty(&bug.plan)
.with_context(|| "should be able to serialize plan")?,
)
.with_context(|| "should be able to write plan file")?;
if let Some(shrunk_plan) = &bug.shrunk_plan {
let shrunk_plan_path = bug_path.join("shrunk.json");
std::fs::write(
&shrunk_plan_path,
serde_json::to_string_pretty(shrunk_plan)
.with_context(|| "should be able to serialize shrunk plan")?,
)
.with_context(|| "should be able to write shrunk plan file")?;
let readable_shrunk_plan_path = bug_path.join("shrunk.sql");
std::fs::write(&readable_shrunk_plan_path, shrunk_plan.to_string())
.with_context(|| "should be able to write readable shrunk plan file")?;
}
let readable_plan_path = bug_path.join("plan.sql");
std::fs::write(&readable_plan_path, bug.plan.to_string())
.with_context(|| "should be able to write readable plan file")?;
let runs_path = bug_path.join("runs.json");
std::fs::write(
&runs_path,
serde_json::to_string_pretty(&bug.runs)
.with_context(|| "should be able to serialize runs")?,
)
.with_context(|| "should be able to write runs file")?;
}
}
Ok(())
}
pub(crate) fn load_bug(&mut self, seed: u64) -> anyhow::Result<LoadedBug> {
let seed_match = self.bugs.get(&seed);
match seed_match {
None => anyhow::bail!("No bugs found for seed {}", seed),
Some(Bug::Unloaded { .. }) => {
let plan =
std::fs::read_to_string(self.path.join(seed.to_string()).join("plan.json"))
.with_context(|| {
format!(
"should be able to read plan file at {}",
self.path.join(seed.to_string()).join("plan.json").display()
)
})?;
let plan: InteractionPlan = serde_json::from_str(&plan)
.with_context(|| "should be able to deserialize plan")?;
let shrunk_plan: Option<String> =
std::fs::read_to_string(self.path.join(seed.to_string()).join("shrunk.json"))
.with_context(|| "should be able to read shrunk plan file")
.and_then(|shrunk| {
serde_json::from_str(&shrunk).map_err(|e| anyhow!("{}", e))
})
.ok();
let shrunk_plan: Option<InteractionPlan> =
shrunk_plan.and_then(|shrunk_plan| serde_json::from_str(&shrunk_plan).ok());
let runs =
std::fs::read_to_string(self.path.join(seed.to_string()).join("runs.json"))
.with_context(|| "should be able to read runs file")
.and_then(|runs| serde_json::from_str(&runs).map_err(|e| anyhow!("{}", e)))
.unwrap_or_default();
let bug = LoadedBug {
seed,
plan: plan.clone(),
runs,
shrunk_plan,
};
self.bugs.insert(seed, Bug::Loaded(bug.clone()));
tracing::debug!("Loaded bug with seed {}", seed);
Ok(bug)
}
Some(Bug::Loaded(bug)) => {
tracing::warn!(
"Bug with seed {} is already loaded, returning the existing plan",
seed
);
Ok(bug.clone())
}
}
}
self.bugs.insert(seed, Some(bug.clone()));
self.bugs.get_mut(&seed).unwrap().as_mut().unwrap()
};
#[expect(dead_code)]
pub(crate) fn mark_successful_run(
&mut self,
seed: u64,
cli_options: &SimulatorCLI,
) -> anyhow::Result<()> {
let bug = self.get_bug(seed);
match bug {
None => {
tracing::debug!("removing bug base entry for {}", seed);
std::fs::remove_dir_all(self.path.join(seed.to_string()))
.with_context(|| "should be able to remove bug directory")?;
}
Some(_) => {
let mut bug = self.load_bug(seed)?;
bug.runs.push(BugRun {
hash: Self::get_current_commit_hash()?,
timestamp: SystemTime::now().into(),
error: None,
cli_options: cli_options.clone(),
shrunk: false,
});
self.bugs.insert(seed, Bug::Loaded(bug.clone()));
// Save the bug to the bug base.
self.save_bug(seed)
.with_context(|| "should be able to save bug")?;
tracing::debug!("Updated bug with seed {}", seed);
}
bug.save_to_path(&path)
}
Ok(())
pub fn get_or_load_bug(&mut self, seed: u64) -> anyhow::Result<Option<&mut Bug>> {
// Check if the bug exists and is loaded
let needs_loading = match self.bugs.get(&seed) {
Some(Some(_)) => false, // Already loaded
Some(None) => true, // Exists but unloaded
None => return Ok(None), // Doesn't exist
};
if needs_loading {
let bug = self.load_bug(seed)?;
self.bugs.insert(seed, Some(bug));
}
pub(crate) fn make_shrunk(
// Now get the mutable reference
Ok(self.bugs.get_mut(&seed).and_then(|opt| opt.as_mut()))
}
pub(crate) fn save_shrunk(
&mut self,
seed: u64,
cli_options: &SimulatorCLI,
shrunk_plan: InteractionPlan,
error: Option<String>,
) -> anyhow::Result<()> {
let mut bug = self.load_bug(seed)?;
bug.shrunk_plan = Some(shrunk_plan);
let path = self.path.clone();
let bug = self
.get_or_load_bug(seed)?
.expect("bug should have been loaded");
bug.runs.push(BugRun {
hash: Self::get_current_commit_hash()?,
timestamp: SystemTime::now().into(),
@@ -391,27 +265,18 @@ impl BugBase {
cli_options: cli_options.clone(),
shrunk: true,
});
self.bugs.insert(seed, Bug::Loaded(bug.clone()));
bug.shrunk_plan = Some(shrunk_plan);
// Save the bug to the bug base.
self.save_bug(seed)
bug.save_to_path(path)
.with_context(|| "should be able to save shrunk bug")?;
Ok(())
}
pub(crate) fn load_bugs(&mut self) -> anyhow::Result<Vec<LoadedBug>> {
let seeds = self.bugs.keys().copied().collect::<Vec<_>>();
seeds
.iter()
.map(|seed| self.load_bug(*seed))
.collect::<Result<Vec<_>, _>>()
}
pub(crate) fn list_bugs(&mut self) -> anyhow::Result<()> {
let bugs = self.load_bugs()?;
for bug in bugs {
println!("seed: {}", bug.seed);
println!("plan: {}", bug.plan.stats());
println!("runs:");
println!(" ------------------");
for run in &bug.runs {

5
simulator/runner/cli.rs
View File

@@ -55,7 +55,7 @@ pub struct SimulatorCLI {
help = "load plan from the bug base",
conflicts_with = "seed"
)]
pub load: Option<String>,
pub load: Option<u64>,
#[clap(
short = 'w',
long,
@@ -200,6 +200,9 @@ pub enum SimulatorCommand {
impl SimulatorCLI {
pub fn validate(&mut self) -> anyhow::Result<()> {
if self.watch {
anyhow::bail!("watch mode is disabled for now");
}
if self.minimum_tests > self.maximum_tests {
tracing::warn!(
"minimum size '{}' is greater than '{}' maximum size, setting both to '{}'",

2
simulator/runner/differential.rs
View File

@@ -8,7 +8,7 @@ use similar_asserts::SimpleDiff;
use sql_generation::model::table::SimValue;
use crate::{
generation::plan::{ConnectionState, InteractionPlanIterator, InteractionPlanState},
model::interactions::{ConnectionState, InteractionPlanIterator, InteractionPlanState},
runner::execution::ExecutionContinuation,
};

2
simulator/runner/doublecheck.rs
View File

@@ -4,7 +4,7 @@ use std::{
};
use crate::{
generation::plan::{ConnectionState, InteractionPlanIterator, InteractionPlanState},
model::interactions::{ConnectionState, InteractionPlanIterator, InteractionPlanState},
runner::execution::ExecutionContinuation,
};

3
simulator/runner/env.rs
View File

@@ -302,8 +302,7 @@ impl SimulatorEnv {
let mut opts = SimulatorOpts {
seed,
ticks: rng
.random_range(cli_opts.minimum_tests as usize..=cli_opts.maximum_tests as usize),
ticks: usize::MAX,
disable_select_optimizer: cli_opts.disable_select_optimizer,
disable_insert_values_select: cli_opts.disable_insert_values_select,
disable_double_create_failure: cli_opts.disable_double_create_failure,

20
simulator/runner/execution.rs
View File

@@ -5,14 +5,14 @@ use tracing::instrument;
use turso_core::{Connection, LimboError, Result, StepResult, Value};
use crate::{
generation::{
Shadow as _,
plan::{
ConnectionState, Interaction, InteractionPlanIterator, InteractionPlanState,
InteractionType, ResultSet,
generation::Shadow as _,
model::{
Query, ResultSet,
interactions::{
ConnectionState, Interaction, InteractionBuilder, InteractionPlanIterator,
InteractionPlanState, InteractionType,
},
},
model::Query,
};
use super::env::{SimConnection, SimulatorEnv};
@@ -199,10 +199,10 @@ pub fn execute_interaction_turso(
stack.push(results);
let query_interaction = Interaction::new(
interaction.connection_index,
InteractionType::Query(query.clone()),
);
let query_interaction = InteractionBuilder::from_interaction(interaction)
.interaction(InteractionType::Query(query.clone()))
.build()
.unwrap();
execute_interaction(env, &query_interaction, stack)?;
}

522
simulator/shrink/plan.rs
View File

@@ -1,31 +1,23 @@
use indexmap::IndexSet;
use sql_generation::model::query::alter_table::{AlterTable, AlterTableType};
use crate::{
SandboxedResult, SimulatorEnv,
generation::{
plan::{InteractionPlan, InteractionType, Interactions, InteractionsType},
property::Property,
model::{
Query,
interactions::{InteractionPlan, InteractionType},
property::PropertyDiscriminants,
},
model::Query,
run_simulation,
runner::execution::Execution,
};
use std::{
collections::HashMap,
num::NonZeroUsize,
ops::Range,
sync::{Arc, Mutex},
};
fn retain_relevant_queries(
extensional_queries: &mut Vec<Query>,
depending_tables: &IndexSet<String>,
) {
extensional_queries.retain(|query| {
query.is_transaction()
|| (!matches!(query, Query::Select(..))
&& query.uses().iter().any(|t| depending_tables.contains(t)))
});
}
impl InteractionPlan {
/// Create a smaller interaction plan by deleting a property
pub(crate) fn shrink_interaction_plan(&self, failing_execution: &Execution) -> InteractionPlan {
@@ -34,54 +26,112 @@ impl InteractionPlan {
// - Shrink properties by removing their extensions, or shrinking their values
let mut plan = self.clone();
let all_interactions = self.interactions_list_with_secondary_index();
let secondary_interactions_index = all_interactions[failing_execution.interaction_index].0;
let all_interactions = self.interactions_list();
let failing_interaction = &all_interactions[failing_execution.interaction_index];
// Index of the parent property where the interaction originated from
let failing_property = &self[secondary_interactions_index];
let mut depending_tables = failing_property.dependencies();
let range = self.find_interactions_range(failing_interaction.id());
{
let mut idx = failing_execution.interaction_index;
loop {
if all_interactions[idx].0 != secondary_interactions_index {
// Stop when we reach a different property
break;
}
match &all_interactions[idx].1.interaction {
// Interactions that are part of the failing overall property
let mut failing_property = all_interactions
[range.start..=failing_execution.interaction_index]
.iter()
.rev();
let mut depending_tables = failing_property
.find_map(|interaction| {
match &interaction.interaction {
InteractionType::Query(query) | InteractionType::FaultyQuery(query) => {
depending_tables = query.dependencies();
break;
}
// Fault does not depend on
InteractionType::Fault(..) => break,
_ => {
// In principle we should never fail this checked_sub.
// But if there is a bug in how we count the secondary index
// we may panic if we do not use a checked_sub.
if let Some(new_idx) = idx.checked_sub(1) {
idx = new_idx;
} else {
tracing::warn!("failed to find error query");
break;
}
Some(query.dependencies())
}
// Fault does not depend on tables
InteractionType::Fault(..) => None,
InteractionType::Assertion(assert) | InteractionType::Assumption(assert) => {
(!assert.tables.is_empty()).then(|| assert.dependencies())
}
_ => None,
}
})
.unwrap_or_else(IndexSet::new);
// Iterate over the rest of the interactions to identify if the depending tables ever changed names
all_interactions[..range.start]
.iter()
.rev()
.for_each(|interaction| match &interaction.interaction {
InteractionType::Query(query)
| InteractionType::FsyncQuery(query)
| InteractionType::FaultyQuery(query) => {
if let Query::AlterTable(AlterTable {
table_name,
alter_table_type: AlterTableType::RenameTo { new_name },
}) = query
{
if depending_tables.contains(new_name)
|| depending_tables.contains(table_name)
{
depending_tables.insert(new_name.clone());
depending_tables.insert(table_name.clone());
}
}
}
_ => {}
});
let before = self.len();
// Remove all properties after the failing one
plan.truncate(secondary_interactions_index + 1);
plan.truncate(failing_execution.interaction_index + 1);
// means we errored in some fault on transaction statement so just maintain the statements from before the failing one
if !depending_tables.is_empty() {
plan.remove_properties(&depending_tables, secondary_interactions_index);
plan.remove_properties(&depending_tables, range);
}
let after = plan.len();
tracing::info!(
"Shrinking interaction plan from {} to {} interactions",
before,
after
);
plan
}
/// Create a smaller interaction plan by deleting a property
pub(crate) fn brute_shrink_interaction_plan(
&self,
result: &SandboxedResult,
env: Arc<Mutex<SimulatorEnv>>,
) -> InteractionPlan {
let failing_execution = match result {
SandboxedResult::Panicked {
error: _,
last_execution: e,
} => e,
SandboxedResult::FoundBug {
error: _,
history: _,
last_execution: e,
} => e,
SandboxedResult::Correct => {
unreachable!("shrink is never called on correct result")
}
};
let mut plan = self.clone();
let all_interactions = self.interactions_list();
let property_id = all_interactions[failing_execution.interaction_index].id();
let before = self.len_properties();
plan.truncate(failing_execution.interaction_index + 1);
// phase 2: shrink the entire plan
plan = Self::iterative_shrink(&plan, failing_execution, result, env, property_id);
let after = plan.len_properties();
tracing::info!(
"Shrinking interaction plan from {} to {} properties",
before,
@@ -91,97 +141,136 @@ impl InteractionPlan {
plan
}
/// shrink a plan by removing one interaction at a time (and its deps) while preserving the error
fn iterative_shrink(
plan: &InteractionPlan,
failing_execution: &Execution,
old_result: &SandboxedResult,
env: Arc<Mutex<SimulatorEnv>>,
failing_property_id: NonZeroUsize,
) -> InteractionPlan {
let mut iter_properties = plan.rev_iter_properties();
let mut ret_plan = plan.clone();
while let Some(property_interactions) = iter_properties.next_property() {
// get the overall property id and try to remove it
// need to consume the iterator, to advance outer iterator
if let Some((_, interaction)) = property_interactions.last()
&& interaction.id() != failing_property_id
{
// try to remove the property
let mut test_plan = ret_plan.clone();
test_plan.remove_property(interaction.id());
if Self::test_shrunk_plan(&test_plan, failing_execution, old_result, env.clone()) {
ret_plan = test_plan;
}
}
}
ret_plan
}
fn test_shrunk_plan(
test_plan: &InteractionPlan,
failing_execution: &Execution,
old_result: &SandboxedResult,
env: Arc<Mutex<SimulatorEnv>>,
) -> bool {
let last_execution = Arc::new(Mutex::new(*failing_execution));
let result = SandboxedResult::from(
std::panic::catch_unwind(|| {
let plan = test_plan.static_iterator();
run_simulation(env.clone(), plan, last_execution.clone())
}),
last_execution,
);
match (old_result, &result) {
(
SandboxedResult::Panicked { error: e1, .. },
SandboxedResult::Panicked { error: e2, .. },
)
| (
SandboxedResult::FoundBug { error: e1, .. },
SandboxedResult::FoundBug { error: e2, .. },
) => e1 == e2,
_ => false,
}
}
/// Remove all properties that do not use the failing tables
fn remove_properties(
&mut self,
depending_tables: &IndexSet<String>,
failing_interaction_index: usize,
failing_interaction_range: Range<usize>,
) {
let mut idx = 0;
// Remove all properties that do not use the failing tables
self.retain_mut(|interactions| {
let retain = if idx == failing_interaction_index {
// First pass - mark indexes that should be retained
let mut retain_map = Vec::with_capacity(self.len());
let mut iter_properties = self.iter_properties();
while let Some(property_interactions) = iter_properties.next_property() {
for (idx, interaction) in property_interactions {
let retain = if failing_interaction_range.end == idx {
true
} else {
let mut has_table = interactions
let is_part_of_property = failing_interaction_range.contains(&idx);
let has_table = interaction
.uses()
.iter()
.any(|t| depending_tables.contains(t));
if has_table {
// will contain extensional queries that reference the depending tables
let mut extensional_queries = Vec::new();
let is_fault = matches!(&interaction.interaction, InteractionType::Fault(..));
let is_transaction = matches!(
&interaction.interaction,
InteractionType::Query(Query::Begin(..))
| InteractionType::Query(Query::Commit(..))
| InteractionType::Query(Query::Rollback(..))
);
let is_pragma = matches!(
&interaction.interaction,
InteractionType::Query(Query::Pragma(..))
);
// Remove the extensional parts of the properties
if let InteractionsType::Property(p) = &mut interactions.interactions {
match p {
Property::InsertValuesSelect { queries, .. }
| Property::DoubleCreateFailure { queries, .. }
| Property::DeleteSelect { queries, .. }
| Property::DropSelect { queries, .. }
| Property::Queries { queries } => {
// Remove placeholder queries
queries.retain(|query| !matches!(query, Query::Placeholder));
extensional_queries.append(queries);
}
Property::AllTableHaveExpectedContent { tables } => {
tables.retain(|table| depending_tables.contains(table));
}
Property::FsyncNoWait { .. } | Property::FaultyQuery { .. } => {}
Property::SelectLimit { .. }
| Property::SelectSelectOptimizer { .. }
| Property::WhereTrueFalseNull { .. }
| Property::UNIONAllPreservesCardinality { .. }
| Property::ReadYourUpdatesBack { .. }
| Property::TableHasExpectedContent { .. } => {}
}
}
// Check again after query clear if the interactions still uses the failing table
has_table = interactions
.uses()
.iter()
.any(|t| depending_tables.contains(t));
// means the queries in the original property are present in the depending tables regardless of the extensional queries
if has_table {
if let Some(queries) = interactions.get_extensional_queries() {
retain_relevant_queries(&mut extensional_queries, depending_tables);
queries.append(&mut extensional_queries);
let skip_interaction = if let Some(property_meta) = interaction.property_meta {
if matches!(
property_meta.property,
PropertyDiscriminants::AllTableHaveExpectedContent
| PropertyDiscriminants::SelectLimit
| PropertyDiscriminants::SelectSelectOptimizer
| PropertyDiscriminants::TableHasExpectedContent
| PropertyDiscriminants::UnionAllPreservesCardinality
| PropertyDiscriminants::WhereTrueFalseNull
) {
// Theses properties only emit select queries, so they can be discarded entirely
true
} else {
property_meta.extension
&& matches!(
&interaction.interaction,
InteractionType::Query(Query::Select(..))
)
}
} else {
// original property without extensional queries does not reference the tables so convert the property to
// `Property::Queries` if `extensional_queries` is not empty
retain_relevant_queries(&mut extensional_queries, depending_tables);
if !extensional_queries.is_empty() {
has_table = true;
*interactions = Interactions::new(
interactions.connection_index,
InteractionsType::Property(Property::Queries {
queries: extensional_queries,
}),
);
}
}
}
let is_fault = matches!(interactions.interactions, InteractionsType::Fault(..));
let is_transaction = matches!(
interactions.interactions,
InteractionsType::Query(Query::Begin(..))
| InteractionsType::Query(Query::Commit(..))
| InteractionsType::Query(Query::Rollback(..))
);
is_fault
|| is_transaction
|| (has_table
&& !matches!(
interactions.interactions,
InteractionsType::Query(Query::Select(_))
| InteractionsType::Property(Property::SelectLimit { .. })
| InteractionsType::Property(
Property::SelectSelectOptimizer { .. }
matches!(
&interaction.interaction,
InteractionType::Query(Query::Select(..))
)
))
};
(is_part_of_property || !skip_interaction)
&& (is_fault || is_transaction || is_pragma || has_table)
};
retain_map.push(retain);
}
}
debug_assert_eq!(self.len(), retain_map.len());
let mut idx = 0;
// Remove all properties that do not use the failing tables
self.retain_mut(|_| {
let retain = retain_map[idx];
idx += 1;
retain
});
@@ -191,23 +280,23 @@ impl InteractionPlan {
// Comprises of idxs of Commit and Rollback intereactions
let mut end_tx_idx: HashMap<usize, Vec<usize>> = HashMap::new();
for (idx, interactions) in self.iter().enumerate() {
match &interactions.interactions {
InteractionsType::Query(Query::Begin(..)) => {
for (idx, interaction) in self.interactions_list().iter().enumerate() {
match &interaction.interaction {
InteractionType::Query(Query::Begin(..)) => {
begin_idx
.entry(interactions.connection_index)
.entry(interaction.connection_index)
.or_insert_with(|| vec![idx]);
}
InteractionsType::Query(Query::Commit(..))
| InteractionsType::Query(Query::Rollback(..)) => {
InteractionType::Query(Query::Commit(..))
| InteractionType::Query(Query::Rollback(..)) => {
let last_begin = begin_idx
.get(&interactions.connection_index)
.get(&interaction.connection_index)
.and_then(|list| list.last())
.unwrap()
+ 1;
if last_begin == idx {
end_tx_idx
.entry(interactions.connection_index)
.entry(interaction.connection_index)
.or_insert_with(|| vec![idx]);
}
}
@@ -241,181 +330,4 @@ impl InteractionPlan {
retain
});
}
/// Create a smaller interaction plan by deleting a property
pub(crate) fn brute_shrink_interaction_plan(
&self,
result: &SandboxedResult,
env: Arc<Mutex<SimulatorEnv>>,
) -> InteractionPlan {
let failing_execution = match result {
SandboxedResult::Panicked {
error: _,
last_execution: e,
} => e,
SandboxedResult::FoundBug {
error: _,
history: _,
last_execution: e,
} => e,
SandboxedResult::Correct => {
unreachable!("shrink is never called on correct result")
}
};
let mut plan = self.clone();
let all_interactions = self.interactions_list_with_secondary_index();
let secondary_interactions_index = all_interactions[failing_execution.interaction_index].0;
{
let mut idx = failing_execution.interaction_index;
loop {
if all_interactions[idx].0 != secondary_interactions_index {
// Stop when we reach a different property
break;
}
match &all_interactions[idx].1.interaction {
// Fault does not depend on
InteractionType::Fault(..) => break,
_ => {
// In principle we should never fail this checked_sub.
// But if there is a bug in how we count the secondary index
// we may panic if we do not use a checked_sub.
if let Some(new_idx) = idx.checked_sub(1) {
idx = new_idx;
} else {
tracing::warn!("failed to find error query");
break;
}
}
}
}
}
let before = self.len();
plan.truncate(secondary_interactions_index + 1);
// phase 1: shrink extensions
for interaction in &mut plan {
if let InteractionsType::Property(property) = &mut interaction.interactions {
match property {
Property::InsertValuesSelect { queries, .. }
| Property::DoubleCreateFailure { queries, .. }
| Property::DeleteSelect { queries, .. }
| Property::DropSelect { queries, .. }
| Property::Queries { queries } => {
let mut temp_plan = InteractionPlan::new_with(
queries
.iter()
.map(|q| {
Interactions::new(
interaction.connection_index,
InteractionsType::Query(q.clone()),
)
})
.collect(),
self.mvcc,
);
temp_plan = InteractionPlan::iterative_shrink(
temp_plan,
failing_execution,
result,
env.clone(),
secondary_interactions_index,
);
//temp_plan = Self::shrink_queries(temp_plan, failing_execution, result, env);
*queries = temp_plan
.into_iter()
.filter_map(|i| match i.interactions {
InteractionsType::Query(q) => Some(q),
_ => None,
})
.collect();
}
Property::WhereTrueFalseNull { .. }
| Property::UNIONAllPreservesCardinality { .. }
| Property::SelectLimit { .. }
| Property::SelectSelectOptimizer { .. }
| Property::FaultyQuery { .. }
| Property::FsyncNoWait { .. }
| Property::ReadYourUpdatesBack { .. }
| Property::TableHasExpectedContent { .. }
| Property::AllTableHaveExpectedContent { .. } => {}
}
}
}
// phase 2: shrink the entire plan
plan = Self::iterative_shrink(
plan,
failing_execution,
result,
env,
secondary_interactions_index,
);
let after = plan.len();
tracing::info!(
"Shrinking interaction plan from {} to {} properties",
before,
after
);
plan
}
/// shrink a plan by removing one interaction at a time (and its deps) while preserving the error
fn iterative_shrink(
mut plan: InteractionPlan,
failing_execution: &Execution,
old_result: &SandboxedResult,
env: Arc<Mutex<SimulatorEnv>>,
secondary_interaction_index: usize,
) -> InteractionPlan {
for i in (0..plan.len()).rev() {
if i == secondary_interaction_index {
continue;
}
let mut test_plan = plan.clone();
test_plan.remove(i);
if Self::test_shrunk_plan(&test_plan, failing_execution, old_result, env.clone()) {
plan = test_plan;
}
}
plan
}
fn test_shrunk_plan(
test_plan: &InteractionPlan,
failing_execution: &Execution,
old_result: &SandboxedResult,
env: Arc<Mutex<SimulatorEnv>>,
) -> bool {
let last_execution = Arc::new(Mutex::new(*failing_execution));
let result = SandboxedResult::from(
std::panic::catch_unwind(|| {
let plan = test_plan.static_iterator();
run_simulation(env.clone(), plan, last_execution.clone())
}),
last_execution,
);
match (old_result, &result) {
(
SandboxedResult::Panicked { error: e1, .. },
SandboxedResult::Panicked { error: e2, .. },
)
| (
SandboxedResult::FoundBug { error: e1, .. },
SandboxedResult::FoundBug { error: e2, .. },
) => e1 == e2,
_ => false,
}
}
}