From 1619285ce023ba266c5b2fa7a966efc709d74b05 Mon Sep 17 00:00:00 2001 From: Bendik Lynghaug Date: Sun, 16 Aug 2026 15:05:01 +0200 Subject: [PATCH] Retire SPECS.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The spec served as the build plan; the built thing now documents itself — man pages for the interface, the test suite for behavior, ADRs in redoal for the design conversation. Removing it beats letting it drift. Co-Authored-By: Claude Fable 5 --- .gitea/workflows/release.yml | 2 +- SPECS.md | 130 ----------------------------------- 2 files changed, 1 insertion(+), 131 deletions(-) delete mode 100644 SPECS.md diff --git a/.gitea/workflows/release.yml b/.gitea/workflows/release.yml index 2ac2938..e7cac90 100644 --- a/.gitea/workflows/release.yml +++ b/.gitea/workflows/release.yml @@ -55,7 +55,7 @@ jobs: cp dist/config.toml "$PKG/config.toml.example" scdoc < dist/varde.8.scd > "$PKG/man/varde.8" scdoc < dist/varde-ctl.1.scd > "$PKG/man/varde-ctl.1" - cp README.md SPECS.md LICENSE-MIT LICENSE-APACHE "$PKG/" + cp README.md LICENSE-MIT LICENSE-APACHE "$PKG/" tar czf "$PKG.tar.gz" "$PKG" sha256sum "$PKG.tar.gz" > "$PKG.tar.gz.sha256" diff --git a/SPECS.md b/SPECS.md deleted file mode 100644 index bad031a..0000000 --- a/SPECS.md +++ /dev/null @@ -1,130 +0,0 @@ -# varde — a system blob mirror daemon on iroh - -> Working name: **varde** (Norwegian: a stone cairn used as a waypoint/beacon). Rename freely; the crate prefix is used consistently below so a find-replace suffices. - -This document is a build brief for Claude Code. Execute the milestones in order. Each milestone must compile, pass tests, and be committed before starting the next. Ask before deviating from the architecture; do not ask about implementation minutiae. - -## 1. Vision - -A small, boring, distro-packageable Linux daemon that owns a content-addressed blob store and mirrors content between consenting peers. Applications talk to it over a unix socket: "add this path", "pin this hash", "materialize hash X at path Y". The daemon handles storage, dedup, verification, LAN peer discovery, and replication policy. Think: what Windows Delivery Optimization is for updates, generalized, open, and built on iroh. - -**Design ethos (non-negotiable):** -- Infrastructure, not product. No GUI, no self-updater, no bundled runtime. Small composable pieces with a stable socket interface, a man page, and systemd units. -- Legible to the network. The daemon never camouflages its traffic. Default posture is LAN-only with zero WAN upload. All background transfer is rate-limited and clearly attributable. -- Consent-tiered. Discovery is open; replication is explicit. Nothing is fetched or served without a standing policy the user created. - -**Non-goals (v1):** gesture/DFT discovery (that arrives later as a separate discovery provider — see §6), WAN swarming, ISP policy channels (ALTO), Windows/macOS support, quotas/multi-tenant accounting. - -## 2. Architecture - -Cargo workspace, three crates: - -``` -varde/ -├── varde-proto # API types: requests, responses, events. serde. No I/O. -├── varde-daemon # the service: store, endpoint, policy engine, socket server -└── varde-ctl # CLI client speaking the socket protocol (also serves as API reference impl) -``` - -**Key dependencies and versions:** -- `iroh-blobs = "0.103"` — the post-rewrite line, now the production line (the old 0.35 pin predated iroh 1.0). Use its persistent fs store (`iroh_blobs::store::fs`). -- `iroh = "1"` — iroh 1.0, stable protocol and API. -- `tokio`, `serde`, `serde_json`, `tracing`, `clap` (ctl only), `anyhow`/`thiserror`. -- `zbus` for NetworkManager metered-status (feature-gated, `default-features` on Linux only). - -**Store layout:** `/var/lib/varde/` in system mode, `$XDG_DATA_HOME/varde/` in user mode. Contains the iroh-blobs store directory, a `meta.redb` (or JSON for MVP) for pins/policies/trust, and the endpoint secret key (`0600`). - -**Materialization:** when a client asks for a hash at a path, export from the store. Attempt reflink first (`FICLONE` ioctl via the `reflink-copy` crate) so btrfs/XFS users get zero-copy; fall back to copy. Never hardlink out of the store (store files must stay immutable). Document that the store dir and materialization targets should share a filesystem for reflink to work. - -## 3. Socket API - -JSON Lines over a unix stream socket. One request per line, one response per line, plus an event subscription mode. No varlink/D-Bus in v1 — keep the dependency surface minimal; the protocol is trivially wrappable later. - -Socket paths: `/run/varde/varde.sock` (system), `$XDG_RUNTIME_DIR/varde.sock` (user). Support systemd socket activation (`sd_notify` + `LISTEN_FDS`) but also plain standalone bind for non-systemd distros. - -Requests (define these as enums in `varde-proto`): - -| Request | Semantics | -|---|---| -| `Add { path, recursive }` | Import file/dir into store. Returns root hash (HashSeq for dirs). | -| `Pin { hash, policy }` | Standing intent: keep this content, fetch if absent, serve to peers per policy. | -| `Unpin { hash }` | Remove intent. GC is a separate explicit op. | -| `Materialize { hash, dest, mode }` | Export to path. `mode: reflink-or-copy \| copy`. | -| `Status { hash? }` | Global or per-hash: have/missing bytes, peers, transfer rates. | -| `List {}` | All pins with policies and completeness. | -| `TicketExport { hash }` / `TicketImport { ticket, pin_policy }` | iroh blob tickets — the v1 out-of-band sharing mechanism. | -| `Push { hash, node_id }` | Hand fully-present content to a trusted peer, unprompted. Mutual consent: sender pushes only to peers it trusts, receiver accepts only from peers it trusts, and pins what it accepted. Delivery is verified via the peer's bitfields before the reply. | -| `Gc {}` | Answers unimplemented since iroh-blobs 0.103: gc runs continuously inside the store (`gc_interval_secs`, default 300), protecting pins via a roots snapshot. | -| `Subscribe {}` | Switch connection to event stream (transfer progress, peer joined, pin complete, push received). | - -Every response carries `{"ok": bool, ...}`. Errors are structured (`code`, `message`), never bare strings. - -## 4. Peer discovery and transfer (v1) - -Two mechanisms only: - -1. **Tickets** — manual, explicit, works today. `varde ctl ticket export ` on machine A, `ticket import` on machine B. -2. **LAN discovery** — iroh's local discovery (mDNS-style). Advertise under a stable, documented service so the daemon is *identifiable* on the network, not anonymous noise. Peers on the same LAN that hold overlapping pins exchange content automatically. - -**Trust model v1:** a peer allowlist keyed by iroh NodeId. `varde ctl peer trust `. Content is only *served* to trusted peers and only *fetched* from trusted peers. An `open-lan` policy flag exists per-pin for deliberately public content but defaults off. All fetched data is BLAKE3-verified by iroh-blobs regardless — trust gates participation, not integrity. - -**Traffic posture:** -- Token-bucket rate limiters on upload and download, configurable, with conservative defaults (e.g. 10 MB/s up on LAN, 0 on WAN). -- Set DSCP CS1/LE on the endpoint socket (best-effort; document that it may require `CAP_NET_ADMIN` or be silently ignored — that's fine). -- LEDBAT is not available off-the-shelf in the Rust QUIC stack; do not attempt it in v1. The rate limiter + DSCP + LAN-only default is the v1 answer to "don't anger anyone." Leave a `transport.rs` seam where scavenger congestion control could later slot in. -- **Metered awareness:** query NetworkManager over D-Bus for the metered flag on the active connection. If metered: no background fetching, no serving, period. Feature-gated so the daemon still builds without D-Bus. - -## 5. systemd and packaging - -Provide in `dist/`: -- `varde.service` + `varde.socket` (system) and user-unit variants. Hardening: `DynamicUser=yes` or dedicated `varde` user, `ProtectSystem=strict`, `StateDirectory=varde`, `RuntimeDirectory=varde`, `NoNewPrivileges=yes`, `RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK`. -- A man page `varde.8` and `varde-ctl.1` (scdoc or raw roff, your choice — scdoc preferred). -- Example config `/etc/varde/config.toml`: store path, socket path, rate limits, discovery on/off, WAN policy. -- A `PKGBUILD` skeleton for Arch (the author's daily driver) — best-effort, clearly marked untested. - -Config precedence: CLI flags > env > config file > defaults. The daemon must run with an empty config. - -## 6. Discovery provider seam (the redoal hook) - -Define a trait in `varde-daemon`: - -```rust -/// A source of content announcements the daemon may subscribe to. -/// v1 ships LanDiscovery + TicketImport. A future provider maps -/// gesture-derived gossip TopicIds (redoal) to announcement streams. -trait DiscoveryProvider { - fn subscribe(&self, topic: TopicKey) -> BoxStream; - fn announce(&self, topic: TopicKey, ann: Announcement) -> Result<()>; -} - -struct Announcement { - root: Hash, // HashSeq root - author: PublicKey, // ed25519, signs the announcement - meta: BTreeMap, - providers: Vec, -} -``` - -Announcements are signed; the daemon indexes all announcements on subscribed topics but only auto-fetches from trusted author keys. This is the consent tiering: discovery open, replication explicit. Do not implement a gossip provider in v1 — just make `LanDiscovery` conform to the trait so the seam is proven. - -## 7. Milestones - -Each milestone = compiling code + tests + a short `docs/milestone-N.md` note on decisions made. - -1. **Skeleton.** Workspace, `varde-proto` types, daemon binds socket, `varde-ctl status` round-trips against an empty daemon. Config loading. Tracing setup. -2. **Store.** iroh-blobs 0.35 fs store wired in. `Add`, `List`, `Materialize` (reflink + fallback), `Gc`. Integration test: add a directory tree, materialize it elsewhere, byte-identical, reflink verified on btrfs (skip-if-unsupported in CI). -3. **Transfer.** iroh endpoint, ticket export/import, `Pin` triggers fetch from ticket-embedded provider. Two-daemon integration test over localhost. -4. **LAN + trust.** Local discovery, peer allowlist, auto-sync of overlapping pins between two trusted daemons. Rate limiting. Event subscription streaming progress. -5. **Citizenship.** Metered detection, DSCP attempt, systemd units + socket activation, man pages, config example, hardening pass, `varde-ctl` polish (human + `--json` output). -6. **Seam proof.** `DiscoveryProvider` trait, LanDiscovery refactored onto it, signed `Announcement` type with verification tests. Write `docs/redoal-integration.md` sketching the gesture-topic provider without implementing it. - -## 8. Testing and quality bar - -- `cargo clippy -- -D warnings`, `cargo fmt --check` in CI (a simple Woodpecker/Gitea Actions yaml in `.ci/`). -- Integration tests spawn real daemons on ephemeral sockets/ports; no mocking iroh. -- Property test: any file round-trips add→materialize bit-identically (proptest, bounded sizes). -- The daemon must survive `kill -9` mid-transfer and resume cleanly on restart (iroh-blobs handles partial state; write a test proving it). - -## 9. Style - -Plain Rust, minimal generics, no async-trait acrobatics where a concrete enum suffices. Errors: `thiserror` in libraries, `anyhow` at binary edges. Every public item in `varde-proto` documented. Comments explain *why*, not *what*. No dependency added without a one-line justification in the commit message.