bl
e256f73439
iroh-blobs 0.35 fs store under <store_dir>/blobs. Add imports files or whole directory trees (as Collections, deterministic order), Materialize exports them with a real FICLONE reflink attempt and streaming-copy fallback, Gc is an explicit mark-and-sweep rooted at the pins. Pins, trusted peers and hash formats persist in meta.json (atomic writes). Store reads run on a LocalPool because iroh-blobs entry readers are not Send. Integration tests drive the real daemon binary: directory round trip byte-identical, gc keeps pinned/drops unpinned, reflink verified on the repo's own filesystem (XFS), plus a 32-case proptest round trip. Dependencies: iroh-blobs =0.35.0 (the store itself; pinned per spec), iroh-io (AsyncSliceReader traits to read store entries), reflink-copy (FICLONE with copy fallback, per spec), proptest (dev-only, round-trip property test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
56 lines
3.1 KiB
Markdown
56 lines
3.1 KiB
Markdown
# Milestone 2 — Store
|
||
|
||
iroh-blobs 0.35 persistent fs store wired in under `<store_dir>/blobs`.
|
||
`Add`, `Pin`/`Unpin` (local intent only — fetch comes with milestone 3),
|
||
`Materialize`, `Status {hash}`, `List`, `Gc` all work end to end.
|
||
|
||
## Decisions
|
||
|
||
- **Pins and formats live in `meta.json`** (spec allows JSON for MVP),
|
||
written atomically via temp file + rename. It records pin policies,
|
||
trusted peers (used from milestone 4), and the *format* of hashes we've
|
||
seen (`raw` vs `hash_seq`) — the bytes of a hash alone don't say whether
|
||
it's a directory root, and materialize needs to know. Unknown hashes
|
||
default to `raw`.
|
||
- **Directories are iroh Collections** (HashSeq with a metadata child),
|
||
imported by walking the tree in sorted order for deterministic roots.
|
||
File contents only; names are relative paths with `/` separators.
|
||
Symlinked directories are rejected (cycle risk), symlinked files are
|
||
followed. Collection entry names are sanitized on export (no `..`, no
|
||
absolute components) so a hostile collection can't escape the target
|
||
directory.
|
||
- **Materialize is our own export loop, not iroh's `export()`**, for one
|
||
reason: honest reflink reporting. iroh's fs export does reflink
|
||
internally but doesn't say whether it happened, and its "copy" mode
|
||
reflinks too, which would make our `mode: copy` a lie. We reflink
|
||
(`reflink-copy` crate → `FICLONE`) straight from the store's
|
||
`data/<hash>.data` file when the blob is complete and file-backed
|
||
(small blobs are inlined in redb and are always streamed), and fall
|
||
back to a chunked streaming copy. Verified against real XFS in the
|
||
test suite: `reflink_when_supported` runs under `CARGO_TARGET_TMPDIR`
|
||
(same filesystem as the repo) and degrades to verifying the copy
|
||
fallback on filesystems without reflink support.
|
||
- **Store reads run on a `LocalPool`.** iroh-blobs entry readers yield
|
||
non-`Send` futures; the daemon uses the same local-pool pattern
|
||
iroh-blobs itself uses for its provider and GC tasks, wrapped in one
|
||
`on_pool` helper.
|
||
- **GC is an explicit one-shot mark and sweep** (the spec's `Gc {}` op),
|
||
not iroh's periodic `gc_run` loop, which only offers a recurring timer.
|
||
Roots are the pins (HashSeq roots expand to their children), plus any
|
||
in-flight temp tags and database tags, so a concurrent `Add` can't be
|
||
swept mid-import. `Add` without `Pin` leaves content GC-able by design —
|
||
add-then-pin is the intended sequence, and GC never runs implicitly.
|
||
- **Property test** (32 cases, 0–64 KiB, spanning the ~16 KiB inline
|
||
threshold) drives the real daemon binary over the socket, not the
|
||
library: add → materialize → bit-identical.
|
||
|
||
## Deviations from the spec
|
||
|
||
- The spec suggested reflink via the `reflink-copy` crate; we do use it,
|
||
but from the store's data file rather than wrapping iroh's export —
|
||
see above. Never hardlinks, store files stay immutable, and reflink
|
||
still requires store and destination on one filesystem (documented).
|
||
- btrfs verification specifically: CI/dev runs use whatever filesystem
|
||
hosts the repo (XFS here, reflink exercised for real); a loopback
|
||
btrfs mount would need root, which integration tests shouldn't assume.
|