varde/SPECS.md

varde — a system blob mirror daemon on iroh

Working name: varde (Norwegian: a stone cairn used as a waypoint/beacon). Rename freely; the crate prefix is used consistently below so a find-replace suffices.

This document is a build brief for Claude Code. Execute the milestones in order. Each milestone must compile, pass tests, and be committed before starting the next. Ask before deviating from the architecture; do not ask about implementation minutiae.

1. Vision

A small, boring, distro-packageable Linux daemon that owns a content-addressed blob store and mirrors content between consenting peers. Applications talk to it over a unix socket: "add this path", "pin this hash", "materialize hash X at path Y". The daemon handles storage, dedup, verification, LAN peer discovery, and replication policy. Think: what Windows Delivery Optimization is for updates, generalized, open, and built on iroh.

Design ethos (non-negotiable):

• Infrastructure, not product. No GUI, no self-updater, no bundled runtime. Small composable pieces with a stable socket interface, a man page, and systemd units.
• Legible to the network. The daemon never camouflages its traffic. Default posture is LAN-only with zero WAN upload. All background transfer is rate-limited and clearly attributable.
• Consent-tiered. Discovery is open; replication is explicit. Nothing is fetched or served without a standing policy the user created.

Non-goals (v1): gesture/DFT discovery (that arrives later as a separate discovery provider — see §6), WAN swarming, ISP policy channels (ALTO), Windows/macOS support, quotas/multi-tenant accounting.

2. Architecture

Cargo workspace, three crates:

varde/
├── varde-proto    # API types: requests, responses, events. serde. No I/O.
├── varde-daemon   # the service: store, endpoint, policy engine, socket server
└── varde-ctl      # CLI client speaking the socket protocol (also serves as API reference impl)

Key dependencies and versions:

• iroh-blobs = "0.35" — pin this. Per upstream, the post-0.35 rewrite is not yet production quality; 0.35 is the recommended production line. Use its persistent fs store (iroh_blobs::store::fs).
• iroh — matching version compatible with iroh-blobs 0.35.
• tokio, serde, serde_json, tracing, clap (ctl only), anyhow/thiserror.
• zbus for NetworkManager metered-status (feature-gated, default-features on Linux only).

Store layout: /var/lib/varde/ in system mode, $XDG_DATA_HOME/varde/ in user mode. Contains the iroh-blobs store directory, a meta.redb (or JSON for MVP) for pins/policies/trust, and the endpoint secret key (0600).

Materialization: when a client asks for a hash at a path, export from the store. Attempt reflink first (FICLONE ioctl via the reflink-copy crate) so btrfs/XFS users get zero-copy; fall back to copy. Never hardlink out of the store (store files must stay immutable). Document that the store dir and materialization targets should share a filesystem for reflink to work.

3. Socket API

JSON Lines over a unix stream socket. One request per line, one response per line, plus an event subscription mode. No varlink/D-Bus in v1 — keep the dependency surface minimal; the protocol is trivially wrappable later.

Socket paths: /run/varde/varde.sock (system), $XDG_RUNTIME_DIR/varde.sock (user). Support systemd socket activation (sd_notify + LISTEN_FDS) but also plain standalone bind for non-systemd distros.

Requests (define these as enums in varde-proto):

Request	Semantics
Add { path, recursive }	Import file/dir into store. Returns root hash (HashSeq for dirs).
Pin { hash, policy }	Standing intent: keep this content, fetch if absent, serve to peers per policy.
Unpin { hash }	Remove intent. GC is a separate explicit op.
Materialize { hash, dest, mode }	Export to path. mode: reflink-or-copy | copy.
Status { hash? }	Global or per-hash: have/missing bytes, peers, transfer rates.
List {}	All pins with policies and completeness.
TicketExport { hash } / TicketImport { ticket, pin_policy }	iroh blob tickets — the v1 out-of-band sharing mechanism.
Gc {}	Drop unpinned blobs.
Subscribe {}	Switch connection to event stream (transfer progress, peer joined, pin complete).

Every response carries {"ok": bool, ...}. Errors are structured (code, message), never bare strings.

4. Peer discovery and transfer (v1)

Two mechanisms only:

1. Tickets — manual, explicit, works today. varde ctl ticket export <hash> on machine A, ticket import on machine B.
2. LAN discovery — iroh's local discovery (mDNS-style). Advertise under a stable, documented service so the daemon is identifiable on the network, not anonymous noise. Peers on the same LAN that hold overlapping pins exchange content automatically.

Trust model v1: a peer allowlist keyed by iroh NodeId. varde ctl peer trust <node-id>. Content is only served to trusted peers and only fetched from trusted peers. An open-lan policy flag exists per-pin for deliberately public content but defaults off. All fetched data is BLAKE3-verified by iroh-blobs regardless — trust gates participation, not integrity.

Traffic posture:

• Token-bucket rate limiters on upload and download, configurable, with conservative defaults (e.g. 10 MB/s up on LAN, 0 on WAN).
• Set DSCP CS1/LE on the endpoint socket (best-effort; document that it may require CAP_NET_ADMIN or be silently ignored — that's fine).
• LEDBAT is not available off-the-shelf in the Rust QUIC stack; do not attempt it in v1. The rate limiter + DSCP + LAN-only default is the v1 answer to "don't anger anyone." Leave a transport.rs seam where scavenger congestion control could later slot in.
• Metered awareness: query NetworkManager over D-Bus for the metered flag on the active connection. If metered: no background fetching, no serving, period. Feature-gated so the daemon still builds without D-Bus.

5. systemd and packaging

Provide in dist/:

• varde.service + varde.socket (system) and user-unit variants. Hardening: DynamicUser=yes or dedicated varde user, ProtectSystem=strict, StateDirectory=varde, RuntimeDirectory=varde, NoNewPrivileges=yes, RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK.
• A man page varde.8 and varde-ctl.1 (scdoc or raw roff, your choice — scdoc preferred).
• Example config /etc/varde/config.toml: store path, socket path, rate limits, discovery on/off, WAN policy.
• A PKGBUILD skeleton for Arch (the author's daily driver) — best-effort, clearly marked untested.

Config precedence: CLI flags > env > config file > defaults. The daemon must run with an empty config.

6. Discovery provider seam (the redoal hook)

Define a trait in varde-daemon:

/// A source of content announcements the daemon may subscribe to.
/// v1 ships LanDiscovery + TicketImport. A future provider maps
/// gesture-derived gossip TopicIds (redoal) to announcement streams.
trait DiscoveryProvider {
    fn subscribe(&self, topic: TopicKey) -> BoxStream<Announcement>;
    fn announce(&self, topic: TopicKey, ann: Announcement) -> Result<()>;
}

struct Announcement {
    root: Hash,           // HashSeq root
    author: PublicKey,    // ed25519, signs the announcement
    meta: BTreeMap<String, String>,
    providers: Vec<NodeAddr>,
}

Announcements are signed; the daemon indexes all announcements on subscribed topics but only auto-fetches from trusted author keys. This is the consent tiering: discovery open, replication explicit. Do not implement a gossip provider in v1 — just make LanDiscovery conform to the trait so the seam is proven.

7. Milestones

Each milestone = compiling code + tests + a short docs/milestone-N.md note on decisions made.

1. Skeleton. Workspace, varde-proto types, daemon binds socket, varde-ctl status round-trips against an empty daemon. Config loading. Tracing setup.
2. Store. iroh-blobs 0.35 fs store wired in. Add, List, Materialize (reflink + fallback), Gc. Integration test: add a directory tree, materialize it elsewhere, byte-identical, reflink verified on btrfs (skip-if-unsupported in CI).
3. Transfer. iroh endpoint, ticket export/import, Pin triggers fetch from ticket-embedded provider. Two-daemon integration test over localhost.
4. LAN + trust. Local discovery, peer allowlist, auto-sync of overlapping pins between two trusted daemons. Rate limiting. Event subscription streaming progress.
5. Citizenship. Metered detection, DSCP attempt, systemd units + socket activation, man pages, config example, hardening pass, varde-ctl polish (human + --json output).
6. Seam proof. DiscoveryProvider trait, LanDiscovery refactored onto it, signed Announcement type with verification tests. Write docs/redoal-integration.md sketching the gesture-topic provider without implementing it.

8. Testing and quality bar

• cargo clippy -- -D warnings, cargo fmt --check in CI (a simple Woodpecker/Gitea Actions yaml in .ci/).
• Integration tests spawn real daemons on ephemeral sockets/ports; no mocking iroh.
• Property test: any file round-trips add→materialize bit-identically (proptest, bounded sizes).
• The daemon must survive kill -9 mid-transfer and resume cleanly on restart (iroh-blobs handles partial state; write a test proving it).

9. Style

Plain Rust, minimal generics, no async-trait acrobatics where a concrete enum suffices. Errors: thiserror in libraries, anyhow at binary edges. Every public item in varde-proto documented. Comments explain why, not what. No dependency added without a one-line justification in the commit message.