Milestone 2: persistent blob store, add/materialize/gc

iroh-blobs 0.35 fs store under <store_dir>/blobs. Add imports files or
whole directory trees (as Collections, deterministic order), Materialize
exports them with a real FICLONE reflink attempt and streaming-copy
fallback, Gc is an explicit mark-and-sweep rooted at the pins. Pins,
trusted peers and hash formats persist in meta.json (atomic writes).
Store reads run on a LocalPool because iroh-blobs entry readers are not
Send. Integration tests drive the real daemon binary: directory round
trip byte-identical, gc keeps pinned/drops unpinned, reflink verified on
the repo's own filesystem (XFS), plus a 32-case proptest round trip.

Dependencies: iroh-blobs =0.35.0 (the store itself; pinned per spec),
iroh-io (AsyncSliceReader traits to read store entries), reflink-copy
(FICLONE with copy fallback, per spec), proptest (dev-only, round-trip
property test).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 20:47:11 +02:00
parent 4e1a95613b
commit e256f73439
12 changed files with 5816 additions and 50 deletions
+55
View File
@@ -0,0 +1,55 @@
# Milestone 2 — Store
iroh-blobs 0.35 persistent fs store wired in under `<store_dir>/blobs`.
`Add`, `Pin`/`Unpin` (local intent only — fetch comes with milestone 3),
`Materialize`, `Status {hash}`, `List`, `Gc` all work end to end.
## Decisions
- **Pins and formats live in `meta.json`** (spec allows JSON for MVP),
written atomically via temp file + rename. It records pin policies,
trusted peers (used from milestone 4), and the *format* of hashes we've
seen (`raw` vs `hash_seq`) — the bytes of a hash alone don't say whether
it's a directory root, and materialize needs to know. Unknown hashes
default to `raw`.
- **Directories are iroh Collections** (HashSeq with a metadata child),
imported by walking the tree in sorted order for deterministic roots.
File contents only; names are relative paths with `/` separators.
Symlinked directories are rejected (cycle risk), symlinked files are
followed. Collection entry names are sanitized on export (no `..`, no
absolute components) so a hostile collection can't escape the target
directory.
- **Materialize is our own export loop, not iroh's `export()`**, for one
reason: honest reflink reporting. iroh's fs export does reflink
internally but doesn't say whether it happened, and its "copy" mode
reflinks too, which would make our `mode: copy` a lie. We reflink
(`reflink-copy` crate → `FICLONE`) straight from the store's
`data/<hash>.data` file when the blob is complete and file-backed
(small blobs are inlined in redb and are always streamed), and fall
back to a chunked streaming copy. Verified against real XFS in the
test suite: `reflink_when_supported` runs under `CARGO_TARGET_TMPDIR`
(same filesystem as the repo) and degrades to verifying the copy
fallback on filesystems without reflink support.
- **Store reads run on a `LocalPool`.** iroh-blobs entry readers yield
non-`Send` futures; the daemon uses the same local-pool pattern
iroh-blobs itself uses for its provider and GC tasks, wrapped in one
`on_pool` helper.
- **GC is an explicit one-shot mark and sweep** (the spec's `Gc {}` op),
not iroh's periodic `gc_run` loop, which only offers a recurring timer.
Roots are the pins (HashSeq roots expand to their children), plus any
in-flight temp tags and database tags, so a concurrent `Add` can't be
swept mid-import. `Add` without `Pin` leaves content GC-able by design —
add-then-pin is the intended sequence, and GC never runs implicitly.
- **Property test** (32 cases, 064 KiB, spanning the ~16 KiB inline
threshold) drives the real daemon binary over the socket, not the
library: add → materialize → bit-identical.
## Deviations from the spec
- The spec suggested reflink via the `reflink-copy` crate; we do use it,
but from the store's data file rather than wrapping iroh's export —
see above. Never hardlinks, store files stay immutable, and reflink
still requires store and destination on one filesystem (documented).
- btrfs verification specifically: CI/dev runs use whatever filesystem
hosts the repo (XFS here, reflink exercised for real); a loopback
btrfs mount would need root, which integration tests shouldn't assume.