aljaz/turso
1
0
Fork 0
mirror of https://github.com/aljazceru/turso.git synced 2025-12-18 17:14:20 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,529 Commits 1 Branch 0 Tags
27fcb81f480372e9d43cb15fdcf824b36bed2f68
Commit Graph

156 Commits

Author SHA1 Message Date
Piotr Sarna
57249f2c94 concurrency test: port to OS threads 
Without mutexes, it makes no sense anymore to use shuttle.
Instead, the test cases just spawn OS threads.
Also, a case with overlapping ids is added, to test whether
transactions read their own writes within the same transaction.
 2023-06-12 14:27:40 +02:00
Piotr Sarna
a93fcdcbcf database: make transaction state atomic 
Without atomic access, we're subject to races when inspecting
whether a transaction just changed its state, e.g. from Preparing
to Committed.
 2023-06-12 13:14:46 +02:00
Piotr Sarna
983544dbfd database: implement missing cases for is_version_visible + tests 
Following the Hekaton paper tables, but also taking into account
that in iteration 0 we're only interested in snapshot isolation,
not serializability.
 2023-06-07 13:58:34 +02:00
Piotr Sarna
b4932340f4 database: add a juicy comment about serializability 
And specifically, the amount of things we don't have implemented
to even think of that. It's mostly about tracking commit dependencies
which allow speculative reads/ignores of certain versions,
as well as making sure that in the commit phase, we validate
visibility of all versions read, as well as that our scans
took into account all data. If some version appeared after the transaction
began, and it was not taken into account during its scans, it is considered
a "phantom", and it invalidates the transaction if we strive for
serializability.
 2023-06-07 11:20:00 +02:00
Piotr Sarna
6d82973359 database: restore a CRUCIAL comment about dropping a tx 
... which stops being correct after lock-free!
 2023-06-06 16:04:28 +02:00
Piotr Sarna
ddbcd9be79 database: bring back dropping unused row versions 2023-06-06 12:47:40 +02:00
Piotr Sarna
625394000e unignore test_dirty_read_deleted 2023-06-06 11:34:34 +02:00
Piotr Sarna
47eb149214 database: drop the mutex 
Without a critical section, we naturally hit a few unimplemented
paths when handling concurrent transactions, which is great news!
Visiting previously impossible paths already proves that lock-free
is able to handle concurrency > 1.
Now, the easy part - fixing all the unimplemented paths and making
the Hekaton implementation 100% foolproof.
 2023-06-06 11:28:27 +02:00
Piotr Sarna
a8faffa9f5 database: migrate txs to SkipMap 2023-06-06 09:43:00 +02:00
Piotr Sarna
fdfc4fd5b4 database: drop RefCell from SkipMap 
not needed, the structure is already Send&Sync
 2023-06-05 13:58:24 +02:00
Piotr Sarna
74a7628a0a mvcc-rs: move database tests to a separate file 
That makes the file more human-readable
 2023-06-05 12:50:35 +02:00
Piotr Sarna
77e88d3f04 fix clippy 2023-06-05 11:47:53 +02:00
Pekka Enberg
51f33919d3 Switch to concurrent SkipMap for row versions (#49) 
Let's switch to concurrent SkipMap as the first small step towards
lockless index...
 2023-05-17 15:31:16 +02:00
Piotr Sarna
ae4cc872b6 treewide: overhaul the API to be sync again 
We dropped all occurrences of Tokio to avoid the cost of allocations
induced by async runtimes.
The only async part of the code is now S3 storage, which is just
wrapped in a futures::executor::block_on()
 2023-05-17 12:39:08 +02:00
Piotr Sarna
a28c41f919 bindings: run recovery on MVCCDatabaseOpen 2023-05-16 14:55:16 +02:00
Piotr Sarna
34d21b0eb7 storage: add S3 storage 
It's now possible to run with S3 persistent storage
if bindings are built with `s3_storage` feature.
Regular S3 credentials need to be set up locally,
and all customary env variables like AWS_SECRET_ACCESS_KEY
and AWS_ACCESS_KEY_ID work just fine.

For local development, one can set MVCCRS_ENDPOINT env variable.
For testing with MinIO, the following setup customarily works:
MVCCRS_ENDPOINT=http://localhost:9000 \
  AWS_SECRET_ACCESS_KEY=minioadmin \
  AWS_ACCESS_KEY_ID=minioadmin \
    ./libsql /tmp/testme.db
 2023-05-16 14:36:58 +02:00
Piotr Sarna
5da87739fa bindings: split transcation begin from insert/read 
libSQL expects to be able to begin/commit a transaction
independently of reading or inserting data.
 2023-05-15 14:33:25 +02:00
Piotr Sarna
8b1ef20c08 treewide: drop storage trait 
We're good with an enum, and async_trait has a runtime cost
we don't like.
 2023-05-15 10:50:47 +02:00
Pekka Enberg
7772c65f8d Switch to Tokio's mutex (#40) 
The async trait wrapper is very expensive because it's doing pin boxing
in the hot path. Switch to Tokio's mutex to get back performance.

Fixes #39
 2023-05-12 19:25:08 +02:00
Piotr Sarna
3236421f77 mvcc: switch to (table_id, row_id) for row ids 
With that, we're able to easily distinguish rows from different
tables.
 2023-05-12 12:15:19 +02:00
Piotr Sarna
a782ae5a0a cursor: add is_empty 2023-05-12 10:09:18 +02:00
Piotr Sarna
d5eec5d528 cursor: add MVCCScanCursorPosition 2023-05-11 14:11:10 +02:00
Piotr Sarna
582bf14934 database: keep rows in BTreeMap 
We need ordering libSQL-side.
 2023-05-11 13:38:38 +02:00
Piotr Sarna
e8bdfc8e7a cursor, read: update pointers to *mut u8 2023-05-11 13:35:19 +02:00
Piotr Sarna
54ee330912 cursor: add closing cursor 
... which also terminates the read transaction open for scanning.
 2023-05-11 12:13:27 +02:00
Piotr Sarna
5bdcfc9924 cursor: handle current() graciously when there's no data 2023-05-11 11:08:43 +02:00
Piotr Sarna
ef097362ff mvcc, bindings: expose a scan cursor 2023-05-10 15:48:46 +02:00
Pekka Enberg
57b5919637 Merge pull request #35 from penberg/recovery 
Initial support for recovery
 2023-05-10 13:28:16 +03:00
Pekka Enberg
0cf7b787fd Merge pull request #36 from penberg/readimpl 
bindings: expose reading from the database
 2023-05-10 13:22:57 +03:00
Piotr Sarna
d047a24a32 bindings: expose reading from the database 
The results are returned as a CString for now.
 2023-05-10 12:09:06 +02:00
Pekka Enberg
08adf2118c Rename mutations.png to transactions.png 2023-05-10 13:01:33 +03:00
Pekka Enberg
ec336d5578 Initial support for recovery 
We have the transaction log in persistent storage so let's implement
recovery by replaying that.
 2023-05-10 12:56:19 +03:00
Pekka Enberg
8c56b381c0 Rename mutations to log records 
The Hekaton paper talks about "log records" so let's just run with that
terminology to avoid confusion.
 2023-05-09 21:32:36 +03:00
Pekka Enberg
cf31de5d41 Fix MVCCDatabaseInsert() type signature 
Values are opaque blobs so use "const void *" for C callers.
 2023-05-09 16:14:28 +03:00
Pekka Enberg
3d0c8a415e Add stdint.h include to mvcc.h 2023-05-09 13:49:31 +03:00
Piotr Sarna
d4da54b10b mvcc: build static library for C bindings 
That's how it's currently consumed by libSQL.
 2023-05-09 10:28:26 +02:00
Pekka Enberg
f47eea1a72 Move DESIGN.md to docs 2023-05-09 11:00:00 +03:00
Pekka Enberg
b9575c4375 Rename database directory to mvcc-rs 2023-05-09 10:56:46 +03:00
Pekka Enberg
41bed41544 Improve logging 2023-05-09 10:51:14 +03:00
Pekka Enberg
5ce2bc41f9 cargo fmt 2023-05-09 10:47:43 +03:00
Pekka Enberg
b2f46e156b Improve C binding error reporting 2023-05-09 10:47:40 +03:00
Pekka Enberg
3ecb0fb2a9 Mark MVCCDatabaseOpen() as unsafe 2023-05-09 10:44:06 +03:00
Pekka Enberg
db1e313aca Silence clippy 2023-05-09 10:43:42 +03:00
Pekka Enberg
1fc99181f1 Update README.md 2023-05-09 10:42:05 +03:00
Pekka Enberg
c94561a646 Add mvcc.h to the tree 2023-05-09 10:41:33 +03:00
Pekka Enberg
779ad3066a Add error codes to C bindings 2023-05-09 10:41:13 +03:00
Pekka Enberg
34a4f1a269 Improve generated C bindings 
Before:

```c
typedef LocalClock Clock;

typedef JsonOnDisk Storage;

typedef DatabaseInner<Clock, Storage> Inner;

typedef Database<Clock, Storage, Mutex<Inner>> Db;

typedef struct {
  Db db;
  Runtime runtime;
} DbContext;

extern "C" {

DbContext *mvccrs_new_database(const char *path);

void mvccrs_free_database(Db *db);

int32_t mvccrs_insert(DbContext *db, uint64_t id, const uint8_t *value_ptr, uintptr_t value_len);

} // extern "C"

```

After:

```c

typedef struct DbContext DbContext;

typedef const DbContext *MVCCDatabaseRef;

extern "C" {

MVCCDatabaseRef MVCCDatabaseOpen(const char *path);

void MVCCDatabaseClose(MVCCDatabaseRef db);

int32_t MVCCDatabaseInsert(MVCCDatabaseRef db, uint64_t id, const uint8_t *value_ptr, uintptr_t value_len);

} // extern "C"

```
 2023-05-09 10:34:11 +03:00
Pekka Enberg
124446f17c Generate C header file on cargo build 2023-05-09 10:34:11 +03:00
Pekka Enberg
d5b96d5edf Move C bindings to separate crate 2023-05-09 10:34:11 +03:00
Piotr Sarna
488201cab2 add c_bindings feature for exposing functions to C 
The library can now be compiled to a static or dynamic
native lib, with very basic functionality exposed.
 2023-05-08 14:20:34 +02:00