0d0906dce4
The current status of the PR is halfway. The new framing of simulation runner where `setup_simulation` is separated from `run_simulation` allows for injecting custom plans easily. The PR is currently missing the functionality to update the `SimulatorEnv` ad hoc from the plan, as the environment tables were typically created during the planning phase. The next steps will be to implement a function `fn mk_env(InteractionPlan, SimulatorEnv) -> SimulatorEnv`, add `--load` flag to the CLI for loading a serialized plan file, making a corresponding environment and running the simulation. We can optionally combine this with a `--save` option, in which we keep a seed-vault as part of limbo simulator, corresponding each seed with its generated plan and save the time to regenerate existing seeds by just loading them into memory. I am curious to hear thoughts on this? Would the maintainers be open to adding such a seed-vault? Do you think the saved time would be worth the complexity of the approach? Reviewed-by: Pere Diaz Bou <pere-altea@homail.com> Closes #720
Limbo Simulator
Limbo simulator uses randomized deterministic simulations to test the Limbo database behaviors.
Each simulations begins with a random configurations;
- the database workload distribution(percentages of reads, writes, deletes...),
- database parameters(page size),
- number of reader or writers, etc.
Based on these parameters, we randomly generate interaction plans. Interaction plans consist of statements/queries, and assertions that will be executed in order. The building blocks of interaction plans are;
- Randomly generated SQL queries satisfying the workload distribution,
- Properties, which contain multiple matching queries with assertions indicating the expected result.
An example of a property is the following:
{
"name": "Read your own writes",
"queries": [
"INSERT INTO t1 (id) VALUES (1)",
"SELECT * FROM t1 WHERE id = 1",
],
"assertions": [
"result.rows.length == 1",
"result.rows[0].id == 1"
]
}
The simulator executes the interaction plans in a loop, and checks the assertions. It can add random queries unrelated to the properties without breaking the property invariants to reach more diverse states and respect the configured workload distribution.
The simulator code is broken into 4 main parts:
- Simulator(main.rs): The main entry point of the simulator. It generates random configurations and interaction plans, and executes them.
- Model(model.rs, model/table.rs, model/query.rs): A simpler model of the database, it contains atomic actions for insertion and selection, we use this model while deciding the next actions.
- Generation(generation.rs, generation/table.rs, generation/query.rs, generation/plan.rs): Random generation functions for the database model and interaction plans.
- Properties(properties.rs): Contains the properties that we want to test.
Running the simulator
To run the simulator, you can use the following command:
cargo run
This prompt (in the future) will invoke a clap command line interface to configure the simulator. For now, the simulator runs with the default configurations changing the main.rs file. If you want to see the logs, you can change the RUST_LOG environment variable.
RUST_LOG=info cargo run --bin limbo_sim
Adding new properties
Todo
Adding new generation functions
Todo
Adding new models
Todo
Coverage with Limbo
Todo
Automatic Compatibility Testing with SQLite
Todo