56 lines
3.1 KiB
Markdown
56 lines
3.1 KiB
Markdown
|
# Milestone 2 — Store
|
|||
|
|
|
|||
|
|
iroh-blobs 0.35 persistent fs store wired in under `<store_dir>/blobs`.
|
|||
|
|
`Add`, `Pin`/`Unpin` (local intent only — fetch comes with milestone 3),
|
|||
|
|
`Materialize`, `Status {hash}`, `List`, `Gc` all work end to end.
|
|||
|
|
|
|||
|
|
## Decisions
|
|||
|
|
|
|||
|
|
- **Pins and formats live in `meta.json`** (spec allows JSON for MVP),
|
|||
|
|
written atomically via temp file + rename. It records pin policies,
|
|||
|
|
trusted peers (used from milestone 4), and the *format* of hashes we've
|
|||
|
|
seen (`raw` vs `hash_seq`) — the bytes of a hash alone don't say whether
|
|||
|
|
it's a directory root, and materialize needs to know. Unknown hashes
|
|||
|
|
default to `raw`.
|
|||
|
|
- **Directories are iroh Collections** (HashSeq with a metadata child),
|
|||
|
|
imported by walking the tree in sorted order for deterministic roots.
|
|||
|
|
File contents only; names are relative paths with `/` separators.
|
|||
|
|
Symlinked directories are rejected (cycle risk), symlinked files are
|
|||
|
|
followed. Collection entry names are sanitized on export (no `..`, no
|
|||
|
|
absolute components) so a hostile collection can't escape the target
|
|||
|
|
directory.
|
|||
|
|
- **Materialize is our own export loop, not iroh's `export()`**, for one
|
|||
|
|
reason: honest reflink reporting. iroh's fs export does reflink
|
|||
|
|
internally but doesn't say whether it happened, and its "copy" mode
|
|||
|
|
reflinks too, which would make our `mode: copy` a lie. We reflink
|
|||
|
|
(`reflink-copy` crate → `FICLONE`) straight from the store's
|
|||
|
|
`data/<hash>.data` file when the blob is complete and file-backed
|
|||
|
|
(small blobs are inlined in redb and are always streamed), and fall
|
|||
|
|
back to a chunked streaming copy. Verified against real XFS in the
|
|||
|
|
test suite: `reflink_when_supported` runs under `CARGO_TARGET_TMPDIR`
|
|||
|
|
(same filesystem as the repo) and degrades to verifying the copy
|
|||
|
|
fallback on filesystems without reflink support.
|
|||
|
|
- **Store reads run on a `LocalPool`.** iroh-blobs entry readers yield
|
|||
|
|
non-`Send` futures; the daemon uses the same local-pool pattern
|
|||
|
|
iroh-blobs itself uses for its provider and GC tasks, wrapped in one
|
|||
|
|
`on_pool` helper.
|
|||
|
|
- **GC is an explicit one-shot mark and sweep** (the spec's `Gc {}` op),
|
|||
|
|
not iroh's periodic `gc_run` loop, which only offers a recurring timer.
|
|||
|
|
Roots are the pins (HashSeq roots expand to their children), plus any
|
|||
|
|
in-flight temp tags and database tags, so a concurrent `Add` can't be
|
|||
|
|
swept mid-import. `Add` without `Pin` leaves content GC-able by design —
|
|||
|
|
add-then-pin is the intended sequence, and GC never runs implicitly.
|
|||
|
|
- **Property test** (32 cases, 0–64 KiB, spanning the ~16 KiB inline
|
|||
|
|
threshold) drives the real daemon binary over the socket, not the
|
|||
|
|
library: add → materialize → bit-identical.
|
|||
|
|
|
|||
|
|
## Deviations from the spec
|
|||
|
|
|
|||
|
|
- The spec suggested reflink via the `reflink-copy` crate; we do use it,
|
|||
|
|
but from the store's data file rather than wrapping iroh's export —
|
|||
|
|
see above. Never hardlinks, store files stay immutable, and reflink
|
|||
|
|
still requires store and destination on one filesystem (documented).
|
|||
|
|
- btrfs verification specifically: CI/dev runs use whatever filesystem
|
|||
|
|
hosts the repo (XFS here, reflink exercised for real); a loopback
|
|||
|
|
btrfs mount would need root, which integration tests shouldn't assume.
|