Commit Graph

1097 Commits

Author SHA1 Message Date
jussisaurio
ccdcf302ca quick fix for #399 2024-11-17 17:06:09 +02:00
jussisaurio
de801f1e25 Merge 'vendor sqlite3-parser (lemon-rs)' from Jussi Saurio
- vendor sqlite3-parser (lemon-rs) to have full control over the parser
- update limbo-core in perf/latency/limbo

Reviewed-by: Pekka Enberg <penberg@iki.fi>

Closes #396
2024-11-16 20:14:11 +02:00
jussisaurio
3cc9d9d79f vendor sqlite3-parser (lemon-rs) 2024-11-16 20:08:59 +02:00
Pekka Enberg
8efeb16b82 Merge 'bindings/wasm: "SELECT 1" benchmark' from Pekka Enberg
This adds a benchmark for "SELECT 1" that essentially just evaluates the
overhead of calling into the library. Looks like WebAssembly adds 2.5x
overhead compared to native code...
```
penberg@vonneumann perf % node perf-limbo.js
cpu: Apple M1
runtime: node v18.19.0 (arm64-darwin)

benchmark      time (avg)             (min … max)       p75       p99      p999
------------------------------------------------- -----------------------------
• Statement
------------------------------------------------- -----------------------------
SELECT 1      771 ns/iter     (624 ns … 2'183 µs)    750 ns  1'085 ns  1'750 ns

summary for Statement
  SELECT 1
penberg@vonneumann perf % node perf-better-sqlite3.js
cpu: Apple M1
runtime: node v18.19.0 (arm64-darwin)

benchmark      time (avg)             (min … max)       p75       p99      p999
------------------------------------------------- -----------------------------
• Statement
------------------------------------------------- -----------------------------
SELECT 1      302 ns/iter       (166 ns … 779 µs)    292 ns    375 ns  1'000 ns

summary for Statement
  SELECT 1
```

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #394
2024-11-16 19:00:39 +02:00
Pekka Enberg
9473668196 core: Bump jsonb dependency
The `fast-float` library has soundness issues, which why `jsonb`
switched to `fast-float2`:

https://github.com/databendlabs/jsonb/issues/68
2024-11-16 18:58:13 +02:00
Pekka Enberg
422dac50f2 Merge 'core: Make mockall a dev dependency' from Pekka Enberg
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #392
2024-11-16 13:24:47 +02:00
Pekka Enberg
85b70e381d core: Make mockall a dev dependency 2024-11-16 13:18:35 +02:00
Pekka Enberg
3442287902 Merge 'bindings/wasm: Fix VFS.sync() function signature' from Pekka Enberg
We don't expect any return value from the underlying `fsync()` so fix
the function signature accordingly.
Fixes the following obscure error:
```
TypeError: Cannot convert undefined to a BigInt
    at wasm://wasm/00942492:wasm-function[1501]:0x1c4dc1
    at wasm://wasm/00942492:wasm-function[694]:0x189586
    at wasm://wasm/00942492:wasm-function[50]:0x143d7
    at wasm://wasm/00942492:wasm-function[60]:0x3f91a
    at new Database (/Users/penberg/src/penberg/limbo/bindings/wasm/pkg/limbo_wasm.js:162:26)
    at file:///Users/penberg/src/penberg/limbo/bindings/wasm/perf/perf-limbo.js:5:12
    at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
    at async loadESM (node:internal/process/esm_loader:34:7)
    at async handleMainPromise (node:internal/modules/run_main:106:12)
```

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #395
2024-11-16 13:10:51 +02:00
Pekka Enberg
60cc8c0347 bindings/wasm: "SELECT 1" benchmark
This adds a benchmark for "SELECT 1" that essentially just evaluates the
overhead of calling into the library. Looks like WebAssembly adds 2.5x
overhead compared to native code...

```
penberg@vonneumann perf % node perf-limbo.js
cpu: Apple M1
runtime: node v18.19.0 (arm64-darwin)

benchmark      time (avg)             (min … max)       p75       p99      p999
------------------------------------------------- -----------------------------
• Statement
------------------------------------------------- -----------------------------
SELECT 1      771 ns/iter     (624 ns … 2'183 µs)    750 ns  1'085 ns  1'750 ns

summary for Statement
  SELECT 1
penberg@vonneumann perf % node perf-better-sqlite3.js
cpu: Apple M1
runtime: node v18.19.0 (arm64-darwin)

benchmark      time (avg)             (min … max)       p75       p99      p999
------------------------------------------------- -----------------------------
• Statement
------------------------------------------------- -----------------------------
SELECT 1      302 ns/iter       (166 ns … 779 µs)    292 ns    375 ns  1'000 ns

summary for Statement
  SELECT 1
```
2024-11-16 13:10:23 +02:00
Pekka Enberg
82ec900b07 bindings/wasm: Fix VFS.sync() function signature
We don't expect any return value from the underlying `fsync()` so fix
the function signature accordingly.

Fixes the following obscure error:

```
TypeError: Cannot convert undefined to a BigInt
    at wasm://wasm/00942492:wasm-function[1501]:0x1c4dc1
    at wasm://wasm/00942492:wasm-function[694]:0x189586
    at wasm://wasm/00942492:wasm-function[50]:0x143d7
    at wasm://wasm/00942492:wasm-function[60]:0x3f91a
    at new Database (/Users/penberg/src/penberg/limbo/bindings/wasm/pkg/limbo_wasm.js:162:26)
    at file:///Users/penberg/src/penberg/limbo/bindings/wasm/perf/perf-limbo.js:5:12
    at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
    at async loadESM (node:internal/process/esm_loader:34:7)
    at async handleMainPromise (node:internal/modules/run_main:106:12)
```
2024-11-16 13:07:32 +02:00
Pere Diaz Bou
0e149ae7dd Merge 'core: Reduce library text size some more' from Pekka Enberg
Apply some more tricks from to reduce text size by 1 MB:
https://github.com/johnthagen/min-sized-rust
Before:
  penberg@vonneumann sqlite3 % ls -lh
../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff   6,7M 16 Mar 09:52
../target/release/liblimbo_sqlite3.a
After:
  penberg@vonneumann sqlite3 % ls -lh
../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff   5,6M 16 Mar 09:50
../target/release/liblimbo_sqlite3.a

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #391
2024-11-16 11:34:16 +01:00
Pekka Enberg
2eb299907f Merge 'core: Make JSON support configurable' from Pekka Enberg
This adds a `json` feature flag, which allows users to disable JSON
support if needed.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #390
2024-11-16 12:14:06 +02:00
Pekka Enberg
8afad9f79b core: Reduce library text size some more
Apply some more tricks from to reduce text size by 1 MB:

https://github.com/johnthagen/min-sized-rust

Before:

  penberg@vonneumann sqlite3 % ls -lh ../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff   6,7M 16 Mar 09:52 ../target/release/liblimbo_sqlite3.a

After:

  penberg@vonneumann sqlite3 % ls -lh ../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff   5,6M 16 Mar 09:50 ../target/release/liblimbo_sqlite3.a
2024-11-16 09:53:24 +02:00
Pekka Enberg
cd5db55cf2 core: Make JSON support configurable
This adds a `json` feature flag, which allows users to disable JSON
support if needed.
2024-11-16 09:49:09 +02:00
Pekka Enberg
17c7e8d346 sqlite3: Disable env_logger default features
Let's keep the dependencies as light-weight as possible.
2024-11-16 09:47:12 +02:00
Pekka Enberg
fe75ed275e Merge 'Bump PyO3 dependency' from Pekka Enberg
Github's bot reports a use-after-free issue so let's bump up the
version:
https://github.com/penberg/limbo/security/dependabot/4

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #388
2024-11-16 09:02:10 +02:00
Pekka Enberg
0e2291a57e Merge 'build: Enable LTO for release builds' from Pekka Enberg
This reduces `liblimbo_sqlite3.a` binary size from 25 MB to 10 MB:
Before:
  $ -lh ../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff    25M 15 Mar 18:36
../target/release/liblimbo_sqlite3.a
After:
  $ penberg@vonneumann sqlite3 % ls -lh
../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff   9,9M 15 Mar 18:59
../target/release/liblimbo_sqlite3.a

Closes #389
2024-11-16 09:01:59 +02:00
Pekka Enberg
c3da75d260 build: Enable LTO for release builds
This reduces `liblimbo_sqlite3.a` binary size from 25 MB to 10 MB:

Before:

  $ -lh ../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff    25M 15 Mar 18:36 ../target/release/liblimbo_sqlite3.a

After:

  $ penberg@vonneumann sqlite3 % ls -lh ../target/release/liblimbo_sqlite3.a
  -rw-r--r--  1 penberg  staff   9,9M 15 Mar 18:59 ../target/release/liblimbo_sqlite3.a
2024-11-15 19:06:17 +02:00
Pekka Enberg
851ea65530 Bump PyO3 dependency
Github's bot reports a use-after-free issue so let's bump up the
version:

https://github.com/penberg/limbo/security/dependabot/4
2024-11-15 16:14:34 +02:00
Pekka Enberg
9d96b22b4a Merge 'core: create databases from limbo' from Pere Diaz Bou
Behold, the database emerges from within.

Reviewed-by: Pekka Enberg <penberg@iki.fi>

Closes #387
2024-11-15 15:55:48 +02:00
Pekka Enberg
44ececce7d github: Fix Pekka's email address 2024-11-15 15:55:33 +02:00
Pekka Enberg
5e1184d559 Update COMPAT.md 2024-11-15 13:49:15 +02:00
Pere Diaz Bou
6cd0f03643 core: create databases from limbo 2024-11-15 12:09:07 +01:00
Pere Diaz Bou
87c80b2a07 use core wal file for wasm 2024-11-14 17:54:20 +01:00
Pere Diaz Bou
4a55e0b26d Merge 'core: drop mutex on contents' from Pere Diaz Bou
There is no need to have mutexes on buffers, we will introduce mutexes
if we want later on a file level to introduce serializability.
This pr also includes a fix in the cacheflush procedure where a wait of
inflight writes weren't waited.

Reviewed-by: Pekka Enberg <penberg@ki.fi>

Closes #384
2024-11-13 19:26:50 +01:00
Pere Diaz Bou
a4b473ba59 fmt 2024-11-13 19:17:49 +01:00
Pere Diaz Bou
c46bd63b5a core: drop mutex on contents
There is no need to have mutexes on buffers, we will introduce mutexes
if we want later on a file level to introduce serializability.
2024-11-13 19:13:15 +01:00
Pere Diaz Bou
71ac1eacb7 Merge 'btree: cursor with lineal stack structure' from Pere Diaz Bou
Removed MemPage from the code in favor of an array to encode the stack
of the cursor. This is both simpler and better in terms of memory
access.
O_DIRECT was removed from WAL file as it introduces alignment
constraints that are too hard to follow in regular appends. Maybe in the
future?

Closes #378
2024-11-13 18:23:31 +01:00
Pere Diaz Bou
2a787aedb5 core: extract page stack and document a bit 2024-11-13 17:13:30 +01:00
Pere Diaz Bou
23923e4175 ignore wal checkpoint test 2024-11-13 14:09:38 +00:00
Pere Diaz Bou
9d7a3e8e60 ignore sequential test 2024-11-13 14:04:28 +00:00
Pere Diaz Bou
dfdd8083e2 remove ignore from wal tests 2024-11-13 13:58:05 +00:00
Pere Diaz Bou
cf86da0ecf remove extra workflow 2024-11-13 13:57:15 +00:00
Pere Diaz Bou
e2276c2e9d O_DIRECT disable on WAL 2024-11-13 13:45:42 +00:00
Pekka Enberg
0f2bc822b4 github: Disable Python requirements check
It's failing pretty often and, therefore, noisy...
2024-11-13 14:54:11 +02:00
Pekka Enberg
b4a9fbfe1e Update CHANGELOG 2024-11-13 14:46:49 +02:00
Pekka Enberg
9628c62f61 Merge 'bindings/wasm: Add Database.prepare() and Statement.all()' from Pekka Enberg
This adds `Database.prepare()` and `Statement.all()` APIs to the Wasm
bindings so that you can actually get something useful out of the
library.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #382
2024-11-13 14:46:13 +02:00
Pere Diaz Bou
94a45eab9e checkpoint inflight 2024-11-13 12:04:54 +00:00
Pere Diaz Bou
cbfb45e55b more trace logs 2024-11-13 11:31:22 +00:00
Pere Diaz Bou
87957bf4f2 checkpoint log state 2024-11-13 11:24:07 +00:00
Pere Diaz Bou
2d03e90840 remove test wal 2024-11-13 11:13:02 +00:00
Pere Diaz Bou
13a95b583f debug logs 2024-11-13 11:03:38 +00:00
Pekka Enberg
b7ca2c9823 bindings/wasm: Add Database.prepare() and Statement.all()
This adds `Database.prepare()` and `Statement.all()` APIs to the Wasm
bindings so that you can actually get something useful out of the
library.
2024-11-13 12:22:35 +02:00
Pere Diaz Bou
97ec01a220 fix missing ' 2024-11-13 10:20:16 +00:00
Pere Diaz Bou
cfb6046717 trace instead of debug on debug 2024-11-13 10:14:12 +00:00
Pekka Enberg
1f228db99f Merge 'Fix Wasm packaging' from Pekka Enberg
The `wasm-pack` too has a known bug where it forgets to add snippets to
`package.json`:
https://github.com/rustwasm/wasm-pack/issues/1206
Let's fix that with some bash magic.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #381
2024-11-13 12:02:20 +02:00
Pere Diaz Bou
6834f11e28 add dbug option rust.yml 2024-11-13 10:01:06 +00:00
Pere Diaz Bou
78118bae81 fmt 2024-11-13 09:34:36 +00:00
Pekka Enberg
40f517b03f Fix Wasm packaging
The `wasm-pack` too has a known bug where it forgets to add snippets to
`package.json`:

https://github.com/rustwasm/wasm-pack/issues/1206

Let's fix that with some bash magic.
2024-11-13 11:28:22 +02:00
Pere Diaz Bou
800354144a remove extra join test 2024-11-12 23:11:04 +00:00