aljaz/turso
1
0
Fork 0
mirror of https://github.com/aljazceru/turso.git synced 2026-02-11 03:04:22 +01:00
Code Issues Packages Projects Releases Wiki Activity

Document persistent storage design (#24)

Browse Source
This commit is contained in:
Pekka Enberg
2023-05-06 21:15:08 +03:00
committed by GitHub
parent eb52f3e1cd
commit aac835fce9
3 changed files with 675 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
core/mvcc/DESIGN.md Normal file
View File

@@ -0,0 +1,19 @@
# Design
## Persistent storage
Persistent storage must implement the `Storage` trait that the MVCC module uses to essentially store a write-ahead log (WAL) of mutations.
Figure 1 shows an example of write-ahead log across three transactions.
The first transaction T0 executes a `INSERT (id) VALUES (1)` statement, which results in a mutation with `id` set to `1`, begin timestamp to 0 (which is the transaction ID) and end timestamp as infinity (meaning the row version is still visible).
The second transaction T1 executes another `INSERT` statement, which adds another mutation to the WAL with `id` set to `2`, begin timesstamp to 1 and end timestamp as infinity, similar to what T0 did.
Finally, a third transaction T2 executes two statements: `DELETE WHERE id = 1` and `INSERT (id) VALUES (3)`. The first one results in a mutation with `id` set to `1` and begin timestamp set to 0 (which is the transaction that created the entry). However, the end timestamp is now set to 2 (the current transaction), which means the entry is now deleted.
The second statement results in an entry in the WAL similar to the `INSERT` statements in T0 and T1.
![Mutations](figures/mutations.png)
<p align="center">
Figure 1. Write-ahead log of mutations across three transactions.
</p>
When MVCC bootstraps or recovers, it simply reads the write-ahead log into the in-memory index, and it's good to go.
If the WAL grows big, we can compact it by dropping all entries that are no longer visible after the the latest transaction.

656
core/mvcc/figures/mutations.excalidraw Normal file
View File

@@ -0,0 +1,656 @@
{
"type": "excalidraw",
"version": 2,
"source": "https://excalidraw.com",
"elements": [
{
"id": "tFvpBUMWe3qPFUTQVV14X",
"type": "text",
"x": 233.14035848761839,
"y": 205.73272444200816,
"width": 278.57781982421875,
"height": 25,
"angle": 0,
"strokeColor": "#087f5b",
"backgroundColor": "#82c91e",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"groupIds": [],
"roundness": null,
"seed": 94988319,
"version": 510,
"versionNonce": 1210831775,
"isDeleted": false,
"boundElements": null,
"updated": 1683370319070,
"link": null,
"locked": false,
"text": "<tx=0, id=1, begin=0, end=∞>",
"fontSize": 20,
"fontFamily": 1,
"textAlign": "left",
"verticalAlign": "top",
"baseline": 18,
"containerId": null,
"originalText": "<tx=0, id=1, begin=0, end=∞>",
"lineHeight": 1.25
},
{
"type": "text",
"version": 515,
"versionNonce": 1881893969,
"isDeleted": false,
"id": "7i88n1PIb89NxUbVQmTTi",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 938.4614491858606,
"y": 311.23272444200813,
"strokeColor": "#0b7285",
"backgroundColor": "#82c91e",
"width": 279.0400085449219,
"height": 25,
"seed": 1123646321,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "<tx=2, id=1, begin=0, end=2>",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "<tx=2, id=1, begin=0, end=2>",
"lineHeight": 1.25,
"baseline": 18
},
{
"type": "text",
"version": 556,
"versionNonce": 153125934,
"isDeleted": false,
"id": "Yh8XLtKqXUUYmcmG4SEXn",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 581.1603475012903,
"y": 256.23272444200813,
"strokeColor": "#e67700",
"backgroundColor": "#82c91e",
"width": 270.71783447265625,
"height": 25,
"seed": 1685524017,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683371076075,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "<tx=1, id=2, begin=1, end=∞>",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "<tx=1, id=2, begin=1, end=∞>",
"lineHeight": 1.25,
"baseline": 18
},
{
"id": "8l0CCJzCAtOLt_2GRcNpa",
"type": "text",
"x": 256.1403584876185,
"y": 409.73272444200813,
"width": 234.41998291015625,
"height": 75,
"angle": 0,
"strokeColor": "#087f5b",
"backgroundColor": "#82c91e",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"groupIds": [],
"roundness": null,
"seed": 583129809,
"version": 570,
"versionNonce": 561756721,
"isDeleted": false,
"boundElements": null,
"updated": 1683370316909,
"link": null,
"locked": false,
"text": "BEGIN\nINSERT (id) VALUEs (1)\nCOMMIT",
"fontSize": 20,
"fontFamily": 1,
"textAlign": "left",
"verticalAlign": "top",
"baseline": 68,
"containerId": null,
"originalText": "BEGIN\nINSERT (id) VALUEs (1)\nCOMMIT",
"lineHeight": 1.25
},
{
"type": "text",
"version": 628,
"versionNonce": 282656095,
"isDeleted": false,
"id": "3m7VluAP5tair6-60b_sp",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 962.0903554358606,
"y": 416.23272444200813,
"strokeColor": "#0b7285",
"backgroundColor": "#82c91e",
"width": 243.91998291015625,
"height": 100,
"seed": 479705617,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "BEGIN\nDELETE WHERE id =1\nINSERT (id) VALUES (3)\nCOMMIT",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "BEGIN\nDELETE WHERE id =1\nINSERT (id) VALUES (3)\nCOMMIT",
"lineHeight": 1.25,
"baseline": 93
},
{
"type": "text",
"version": 574,
"versionNonce": 1128746001,
"isDeleted": false,
"id": "Z-Mh1kti2oC6sIMnuGluo",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 613.0903554358607,
"y": 417.23272444200813,
"strokeColor": "#e67700",
"backgroundColor": "#82c91e",
"width": 243.239990234375,
"height": 75,
"seed": 580440625,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "BEGIN\nINSERT (id) VALUEs (2)\nCOMMIT",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "BEGIN\nINSERT (id) VALUEs (2)\nCOMMIT",
"lineHeight": 1.25,
"baseline": 68
},
{
"type": "line",
"version": 1502,
"versionNonce": 1835608607,
"isDeleted": false,
"id": "VuJNZCgz1Y0WEWwug7pGk",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"angle": 0,
"x": 226.3083636621349,
"y": 173.11701218356845,
"strokeColor": "#000000",
"backgroundColor": "transparent",
"width": 20.336010349032712,
"height": 203.23377930246647,
"seed": 1879839231,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"startBinding": null,
"endBinding": null,
"lastCommittedPoint": null,
"startArrowhead": null,
"endArrowhead": null,
"points": [
[
0,
0
],
[
-20.264781987976257,
-0.0011773927935071482
],
[
-20.336010349032712,
203.23260190967298
],
[
-0.07239358683375485,
203.135377672515
]
]
},
{
"type": "line",
"version": 1755,
"versionNonce": 1487752017,
"isDeleted": false,
"id": "GpZg3Rw4Hszxzxf38Q4Hn",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"angle": 3.141592653589793,
"x": 539.3083636621348,
"y": 178.11701218356845,
"strokeColor": "#000000",
"backgroundColor": "transparent",
"width": 20.336010349032712,
"height": 203.23377930246647,
"seed": 470135121,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"startBinding": null,
"endBinding": null,
"lastCommittedPoint": null,
"startArrowhead": null,
"endArrowhead": null,
"points": [
[
0,
0
],
[
-20.264781987976257,
-0.0011773927935071482
],
[
-20.336010349032712,
203.23260190967298
],
[
-0.07239358683375485,
203.135377672515
]
]
},
{
"type": "text",
"version": 528,
"versionNonce": 1276939839,
"isDeleted": false,
"id": "AGEyNvBxBm2cwm1WRW8n8",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 576.6403584876185,
"y": 210.23272444200816,
"strokeColor": "#087f5b",
"backgroundColor": "#82c91e",
"width": 278.57781982421875,
"height": 25,
"seed": 877528401,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "<tx=0, id=1, begin=0, end=∞>",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "<tx=0, id=1, begin=0, end=∞>",
"lineHeight": 1.25,
"baseline": 18
},
{
"type": "line",
"version": 1557,
"versionNonce": 773679889,
"isDeleted": false,
"id": "Q8E0gAcLvq6VXqMDZhLdA",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"angle": 0,
"x": 581.8083636621351,
"y": 177.61701218356845,
"strokeColor": "#000000",
"backgroundColor": "transparent",
"width": 20.336010349032712,
"height": 203.23377930246647,
"seed": 153279217,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"startBinding": null,
"endBinding": null,
"lastCommittedPoint": null,
"startArrowhead": null,
"endArrowhead": null,
"points": [
[
0,
0
],
[
-20.264781987976257,
-0.0011773927935071482
],
[
-20.336010349032712,
203.23260190967298
],
[
-0.07239358683375485,
203.135377672515
]
]
},
{
"type": "line",
"version": 1810,
"versionNonce": 1561283199,
"isDeleted": false,
"id": "uhh3ZkPO6bwwf0-AI8syI",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"angle": 3.141592653589793,
"x": 894.8083636621349,
"y": 182.61701218356845,
"strokeColor": "#000000",
"backgroundColor": "transparent",
"width": 20.336010349032712,
"height": 203.23377930246647,
"seed": 315380945,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"startBinding": null,
"endBinding": null,
"lastCommittedPoint": null,
"startArrowhead": null,
"endArrowhead": null,
"points": [
[
0,
0
],
[
-20.264781987976257,
-0.0011773927935071482
],
[
-20.336010349032712,
203.23260190967298
],
[
-0.07239358683375485,
203.135377672515
]
]
},
{
"type": "text",
"version": 575,
"versionNonce": 910156017,
"isDeleted": false,
"id": "jI5YKyaOdGYYKiBWZmCMs",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 929.6403584876182,
"y": 215.23272444200813,
"strokeColor": "#087f5b",
"backgroundColor": "#82c91e",
"width": 278.57781982421875,
"height": 25,
"seed": 121503167,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "<tx=0, id=1, begin=0, end=∞>",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "<tx=0, id=1, begin=0, end=∞>",
"lineHeight": 1.25,
"baseline": 18
},
{
"type": "line",
"version": 1604,
"versionNonce": 19920575,
"isDeleted": false,
"id": "QqIk7VTnRWYq499wkttvv",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"angle": 0,
"x": 934.8083636621348,
"y": 182.61701218356842,
"strokeColor": "#000000",
"backgroundColor": "transparent",
"width": 20.336010349032712,
"height": 203.23377930246647,
"seed": 2012037663,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"startBinding": null,
"endBinding": null,
"lastCommittedPoint": null,
"startArrowhead": null,
"endArrowhead": null,
"points": [
[
0,
0
],
[
-20.264781987976257,
-0.0011773927935071482
],
[
-20.336010349032712,
203.23260190967298
],
[
-0.07239358683375485,
203.135377672515
]
]
},
{
"type": "line",
"version": 1857,
"versionNonce": 1660885169,
"isDeleted": false,
"id": "gk89VsYpnf9Jby9KEUBd3",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 0,
"opacity": 100,
"angle": 3.141592653589793,
"x": 1247.808363662135,
"y": 187.61701218356842,
"strokeColor": "#000000",
"backgroundColor": "transparent",
"width": 20.336010349032712,
"height": 203.23377930246647,
"seed": 509453887,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370316909,
"link": null,
"locked": false,
"startBinding": null,
"endBinding": null,
"lastCommittedPoint": null,
"startArrowhead": null,
"endArrowhead": null,
"points": [
[
0,
0
],
[
-20.264781987976257,
-0.0011773927935071482
],
[
-20.336010349032712,
203.23260190967298
],
[
-0.07239358683375485,
203.135377672515
]
]
},
{
"type": "text",
"version": 620,
"versionNonce": 1588681010,
"isDeleted": false,
"id": "a1c-iZI0SafCiy0u4xieZ",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 934.3714375891809,
"y": 261.23272444200813,
"strokeColor": "#e67700",
"backgroundColor": "#82c91e",
"width": 270.71783447265625,
"height": 25,
"seed": 1742829553,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683371080181,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "<tx=1, id=2, begin=1, end=∞>",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "<tx=1, id=2, begin=1, end=∞>",
"lineHeight": 1.25,
"baseline": 18
},
{
"type": "text",
"version": 564,
"versionNonce": 1968863633,
"isDeleted": false,
"id": "hdhhgp5nA06o5EcSgHQE8",
"fillStyle": "hachure",
"strokeWidth": 1,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"angle": 0,
"x": 937.6203542151575,
"y": 354.23272444200813,
"strokeColor": "#0b7285",
"backgroundColor": "#82c91e",
"width": 287.73785400390625,
"height": 25,
"seed": 309558367,
"groupIds": [],
"roundness": null,
"boundElements": [],
"updated": 1683370363648,
"link": null,
"locked": false,
"fontSize": 20,
"fontFamily": 1,
"text": "<tx=2, id=3, begin=2, end=∞>",
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "<tx=2, id=3, begin=2, end=∞>",
"lineHeight": 1.25,
"baseline": 18
}
],
"appState": {
"gridSize": null,
"viewBackgroundColor": "#ffffff"
},
"files": {}
}

BIN
core/mvcc/figures/mutations.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 304 KiB