bl/varde
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Migrate to iroh 1.0 and iroh-blobs 0.103

Browse Source 
The 0.35 pin's rationale ("the post-0.35 rewrite is not yet production
quality") expired when iroh hit 1.0 in June 2026: the rewrite line
(0.103) is now the production line and the only one receiving fixes.

The rewrite replaced wrappable store traits with an irpc-based API, so
the seams moved:

- shaped.rs: the 440-line ShapedStore trait wrapper becomes a provider
  event handler. Trust gating (untrusted peers see only openly-served
  hashes) now intercepts requests before any bytes move; upload rate
  limiting rides the provider's Throttle hook; upload progress events
  come from per-request update streams. The TokenBucket is unchanged.
- store.rs: FsStore's API handle replaces the store traits, and the
  LocalPool machinery for non-Send futures is gone. GC is now the
  store's built-in periodic mark-and-sweep, fed by a pin-roots snapshot
  via the protect callback (new config knob gc_interval_secs, default
  300); the on-demand Gc request answers Unimplemented, and the gc
  conformance test polls the sweep instead.
- transfer.rs: BlobsProtocol + Router replace handle_connection, the
  new multi-provider Downloader replaces the old queue, and mdns
  discovery moved to the iroh-mdns-address-lookup crate (it left iroh
  core in 1.0). Ticket-embedded provider addresses feed a MemoryLookup
  address book. Endpoint presets: Minimal (LAN-only default) or N0
  (wan_upload), preserving the old relay posture.
- Node* became Endpoint* throughout; announcement signatures use iroh's
  own Signature type (ed25519-dalek dep dropped); iroh-io dropped.

Known regression: max_download_bytes_per_sec is currently not enforced
— download shaping rode the old store's batch writer and 0.103's
downloader has no equivalent seam yet.

Announcement wire format note: EndpointAddr serializes differently than
NodeAddr, so pre- and post-migration daemons won't parse each other's
LAN announcements. Announcements are live-only, nothing stored breaks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Bendik Lynghaug
2026-08-16 14:03:15 +02:00
parent a2d150225a
commit ad69f7d884
14 changed files with 1807 additions and 2248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
varde-daemon/src/transfer.rs
+129 -184
View File
@@ -1,25 +1,20 @@
//! iroh endpoint, blob provider and downloader.
//!
//! This module is the transport seam: rate limiting lives in
//! [`crate::shaped::ShapedStore`] wired up here, and a scavenger
//! congestion controller would later slot in at the same place without
//! touching the daemon logic above.
//! This module is the transport seam: trust gating and rate limiting
//! live in [`crate::shaped`]'s provider event handler wired up here, and
//! a scavenger congestion controller would later slot in at the same
//! place without touching the daemon logic above.
use std::path::Path;
use std::sync::Arc;
use anyhow::{Context, Result};
use iroh::discovery::mdns::MdnsDiscovery;
use iroh::endpoint::Connection;
use iroh::protocol::{ProtocolHandler, Router};
use iroh::{Endpoint, NodeAddr, NodeId, RelayMode, SecretKey};
use iroh_blobs::downloader::{DownloadRequest, Downloader};
use iroh_blobs::get::db::DownloadProgress;
use iroh_blobs::provider::{handle_connection, CustomEventSender, EventSender};
use iroh_blobs::util::local_pool::LocalPoolHandle;
use iroh_blobs::util::progress::AsyncChannelProgressSender;
use iroh_blobs::{BlobFormat, Hash, HashAndFormat};
use n0_future::boxed::BoxFuture;
use iroh::address_lookup::memory::MemoryLookup;
use iroh::protocol::Router;
use iroh::{Endpoint, EndpointAddr, EndpointId, RelayMode, SecretKey};
use iroh_blobs::api::downloader::{DownloadOptions, Downloader, Shuffled, SplitStrategy};
use iroh_blobs::{BlobFormat, BlobsProtocol, Hash, HashAndFormat};
use iroh_mdns_address_lookup::{DiscoveryEvent, MdnsAddressLookup};
use n0_future::StreamExt;
use tokio::sync::{broadcast, mpsc};
use tracing::{debug, info, warn};
@@ -28,7 +23,7 @@ use varde_proto::{Direction, Event};
use crate::config::Config;
use crate::meta::Meta;
use crate::metered::MeteredState;
use crate::shaped::{ServeFilter, ShapedStore, TokenBucket};
use crate::shaped::{Gate, ServeFilter, TokenBucket};
use crate::store::BlobStore;
/// The network side of the daemon: one iroh endpoint serving the blob
@@ -39,71 +34,13 @@ pub struct Transfer {
endpoint: Endpoint,
router: Router,
downloader: Downloader,
/// Address book for provider addresses learned out-of-band (ticket
/// imports): the downloader dials by endpoint id and resolves
/// addresses through the endpoint's address lookup services.
known_addrs: MemoryLookup,
events: broadcast::Sender<Event>,
metered: MeteredState,
discovery_rx: std::sync::Mutex<Option<mpsc::Receiver<NodeId>>>,
}
/// Serves blob requests with trust gating: trusted peers see the whole
/// store, untrusted peers see only the openly-served subset (and get
/// "not found" for the rest). With serving disabled (upload cap 0) every
/// incoming connection is closed immediately.
#[derive(Debug, Clone)]
struct GatedProvider {
full_view: ShapedStore,
open_view: ShapedStore,
meta: Arc<Meta>,
events: EventSender,
pool: LocalPoolHandle,
serving_enabled: bool,
metered: MeteredState,
}
impl ProtocolHandler for GatedProvider {
fn accept(&self, connection: Connection) -> BoxFuture<Result<()>> {
let this = self.clone();
Box::pin(async move {
let remote = connection.remote_node_id()?;
if !this.serving_enabled || this.metered.is_metered() {
debug!(%remote, "serving disabled or metered, closing connection");
connection.close(0u32.into(), b"serving disabled");
return Ok(());
}
let trusted = this.meta.is_trusted(&remote.to_string());
debug!(%remote, trusted, "incoming blobs connection");
if trusted {
handle_connection(connection, this.full_view, this.events, this.pool).await;
} else {
handle_connection(connection, this.open_view, this.events, this.pool).await;
}
Ok(())
})
}
}
/// Forwards provider transfer events into the daemon event stream.
#[derive(Debug)]
struct UploadEvents(broadcast::Sender<Event>);
impl CustomEventSender for UploadEvents {
fn send(&self, event: iroh_blobs::provider::Event) -> BoxFuture<()> {
self.try_send(event);
Box::pin(async {})
}
fn try_send(&self, event: iroh_blobs::provider::Event) {
if let iroh_blobs::provider::Event::TransferProgress {
hash, end_offset, ..
} = event
{
let _ = self.0.send(Event::TransferProgress {
hash: hash.to_hex().to_string(),
direction: Direction::Upload,
bytes_done: end_offset,
bytes_total: None,
});
}
}
discovery_rx: std::sync::Mutex<Option<mpsc::Receiver<EndpointId>>>,
}
impl Transfer {
@@ -117,60 +54,71 @@ impl Transfer {
store: &BlobStore,
meta: Arc<Meta>,
open_filter: ServeFilter,
pool: LocalPoolHandle,
events: broadcast::Sender<Event>,
metered: MeteredState,
) -> Result<Transfer> {
let secret = load_or_create_secret(&config.store_dir.join("secret.key"))?;
let mut builder = Endpoint::builder().secret_key(secret.clone());
if !config.wan_upload {
builder = builder.relay_mode(RelayMode::Disabled);
}
// With wan_upload the n0 defaults apply (their relays and DNS
// lookup, matching the pre-1.0 default relay mode); otherwise the
// endpoint gets no external services at all.
let builder = if config.wan_upload {
Endpoint::builder(iroh::endpoint::presets::N0)
} else {
Endpoint::builder(iroh::endpoint::presets::Minimal).relay_mode(RelayMode::Disabled)
};
let endpoint = builder
.secret_key(secret.clone())
.bind()
.await
.context("binding iroh endpoint")?;
info!(endpoint_id = %endpoint.id(), discovery = config.discovery, "endpoint up");
let known_addrs = MemoryLookup::with_provenance("varde-ticket");
endpoint
.address_lookup()
.context("endpoint address lookup services")?
.add(known_addrs.clone());
// LAN discovery: advertise under iroh's mDNS-style local
// discovery service (identifiable `_iroh` mdns records, not
// anonymous noise). Discovered node ids stream to the daemon,
// anonymous noise). Discovered endpoint ids stream to the daemon,
// which decides what to do based on trust.
let (discovery_tx, discovery_rx) = mpsc::channel(64);
if config.discovery {
let mdns =
MdnsDiscovery::new(secret.public()).context("starting mdns local discovery")?;
if let Some(mut stream) = iroh::discovery::Discovery::subscribe(&mdns) {
tokio::spawn(async move {
while let Some(item) = stream.next().await {
if discovery_tx.send(item.node_id()).await.is_err() {
break;
}
let mdns = MdnsAddressLookup::builder()
.build(endpoint.id())
.context("starting mdns local discovery")?;
let mut sightings = mdns.subscribe().await;
tokio::spawn(async move {
while let Some(event) = sightings.next().await {
let DiscoveryEvent::Discovered { endpoint_info, .. } = event else {
continue;
};
if discovery_tx.send(endpoint_info.endpoint_id).await.is_err() {
break;
}
});
}
builder = builder.discovery(Box::new(mdns));
}
});
endpoint
.address_lookup()
.context("endpoint address lookup services")?
.add(mdns);
}
let endpoint = builder.bind().await.context("binding iroh endpoint")?;
info!(node_id = %endpoint.node_id(), discovery = config.discovery, "endpoint up");
// Best-effort low-priority marking of our UDP traffic.
let (v4, v6) = endpoint.bound_sockets();
let bound: Vec<std::net::SocketAddr> = std::iter::once(v4).chain(v6).collect();
crate::dscp::mark_endpoint_sockets(&bound);
crate::dscp::mark_endpoint_sockets(&endpoint.bound_sockets());
let upload_bucket = TokenBucket::new(config.max_upload_bytes_per_sec);
let download_bucket = TokenBucket::new(config.max_download_bytes_per_sec);
let shaped = ShapedStore::new(store.inner().clone(), upload_bucket, download_bucket);
let provider_events = EventSender::new(Some(Arc::new(UploadEvents(events.clone()))));
let provider = GatedProvider {
full_view: shaped.clone(),
open_view: shaped.filtered(open_filter),
let provider_events = crate::shaped::provider_events(Gate {
meta,
events: provider_events,
pool: pool.clone(),
open_filter,
upload: TokenBucket::new(config.max_upload_bytes_per_sec),
serving_enabled: config.max_upload_bytes_per_sec > 0,
metered: metered.clone(),
};
events: events.clone(),
});
let provider = BlobsProtocol::new(store.api(), Some(provider_events));
let downloader = Downloader::new(shaped, endpoint.clone(), pool);
let downloader = store.api().downloader(&endpoint);
let router = Router::builder(endpoint.clone())
.accept(iroh_blobs::ALPN, provider)
.spawn();
@@ -179,15 +127,16 @@ impl Transfer {
endpoint,
router,
downloader,
known_addrs,
events,
metered,
discovery_rx: std::sync::Mutex::new(Some(discovery_rx)),
})
}
/// Our stable node id.
/// Our stable endpoint id.
pub fn node_id(&self) -> String {
self.endpoint.node_id().to_string()
self.endpoint.id().to_string()
}
/// The endpoint's identity key (also signs our announcements).
@@ -195,68 +144,89 @@ impl Transfer {
self.endpoint.secret_key().clone()
}
/// Take the stream of locally discovered node ids. Yields each
/// Take the stream of locally discovered endpoint ids. Yields each
/// sighting (mdns re-announces periodically); the daemon dedupes.
/// Callable once; returns None afterwards or with discovery off.
pub fn take_discovery(&self) -> Option<mpsc::Receiver<NodeId>> {
pub fn take_discovery(&self) -> Option<mpsc::Receiver<EndpointId>> {
self.discovery_rx.lock().expect("discovery lock").take()
}
/// Produce a ticket for out-of-band sharing of `content`.
pub async fn export_ticket(&self, content: HashAndFormat) -> Result<String> {
let addr = self
.endpoint
.node_addr()
.await
.context("waiting for endpoint address")?;
let ticket = iroh_blobs::ticket::BlobTicket::new(addr, content.hash, content.format)?;
let addr = self.endpoint.addr();
anyhow::ensure!(
!addr.addrs.is_empty(),
"endpoint has no dialable addresses yet"
);
let ticket = iroh_blobs::ticket::BlobTicket::new(addr, content.hash, content.format);
Ok(ticket.to_string())
}
/// Queue a background fetch of `content` from `providers`. Emits
/// progress events and [`Event::PinComplete`] when fully verified.
pub fn spawn_fetch(&self, content: HashAndFormat, providers: Vec<NodeAddr>) {
pub fn spawn_fetch(&self, content: HashAndFormat, providers: Vec<EndpointAddr>) {
if self.metered.is_metered() {
// The pin stays recorded; auto-sync retries once unmetered.
info!(hash = %content.hash.to_hex(), "metered connection: fetch deferred");
return;
}
// The downloader dials by EndpointId; teach the endpoint any
// direct addresses we know (ticket-embedded ones). Id-only
// provider entries resolve via discovery instead.
let mut ids = Vec::with_capacity(providers.len());
for provider in providers {
ids.push(provider.id);
if !provider.addrs.is_empty() {
self.known_addrs.add_endpoint_info(provider);
}
}
let downloader = self.downloader.clone();
let events = self.events.clone();
let endpoint = self.endpoint.clone();
tokio::spawn(async move {
// The downloader dials by NodeId; the endpoint must be taught
// any direct addresses we know (ticket-embedded ones), or the
// dial fails with "no addressing information". Id-only
// provider entries resolve via discovery instead.
for provider in &providers {
if !provider.direct_addresses.is_empty() || provider.relay_url.is_some() {
if let Err(e) = endpoint.add_node_addr(provider.clone()) {
debug!(node = %provider.node_id, error = %e, "adding provider address");
let progress = downloader.download_with_opts(DownloadOptions::new(
content,
Shuffled::new(ids),
SplitStrategy::None,
));
let mut stream = match progress.stream().await {
Ok(stream) => stream,
Err(e) => {
warn!(hash = %content.hash.to_hex(), error = %e, "fetch failed to start");
return;
}
};
use iroh_blobs::api::downloader::DownloadProgressItem;
let mut failed = false;
while let Some(item) = stream.next().await {
match item {
DownloadProgressItem::Progress(bytes_done) => {
let _ = events.send(Event::TransferProgress {
hash: content.hash.to_hex().to_string(),
direction: Direction::Download,
bytes_done,
bytes_total: None,
});
}
DownloadProgressItem::ProviderFailed { id, .. } => {
debug!(hash = %content.hash.to_hex(), provider = %id, "provider failed");
}
DownloadProgressItem::Error(e) => {
warn!(hash = %content.hash.to_hex(), error = %e, "fetch failed");
failed = true;
}
DownloadProgressItem::DownloadError => {
warn!(hash = %content.hash.to_hex(), "fetch failed");
failed = true;
}
_ => {}
}
}
let (progress_tx, progress_rx) = async_channel::bounded(64);
tokio::spawn(forward_download_progress(
content.hash,
progress_rx,
events.clone(),
));
let request = DownloadRequest::new(content, providers)
.progress_sender(AsyncChannelProgressSender::new(progress_tx));
let handle = downloader.queue(request).await;
match handle.await {
Ok(_stats) => {
info!(hash = %content.hash.to_hex(), "fetch complete");
let _ = events.send(Event::PinComplete {
hash: content.hash.to_hex().to_string(),
});
}
Err(e) => {
warn!(hash = %content.hash.to_hex(), error = %e, "fetch failed");
}
if !failed {
info!(hash = %content.hash.to_hex(), "fetch complete");
let _ = events.send(Event::PinComplete {
hash: content.hash.to_hex().to_string(),
});
}
});
}
@@ -270,37 +240,12 @@ impl Transfer {
}
}
/// Translate iroh-blobs download progress into daemon transfer events.
async fn forward_download_progress(
root: Hash,
rx: async_channel::Receiver<DownloadProgress>,
events: broadcast::Sender<Event>,
) {
let mut current_total = None;
while let Ok(progress) = rx.recv().await {
match progress {
DownloadProgress::Found { size, .. } => {
current_total = Some(size);
}
DownloadProgress::Progress { offset, .. } => {
let _ = events.send(Event::TransferProgress {
hash: root.to_hex().to_string(),
direction: Direction::Download,
bytes_done: offset,
bytes_total: current_total,
});
}
DownloadProgress::AllDone(_) | DownloadProgress::Abort(_) => break,
_ => {}
}
}
}
/// Parse a ticket string into (root, format, provider address).
pub fn parse_ticket(ticket: &str) -> Result<(Hash, BlobFormat, NodeAddr), String> {
pub fn parse_ticket(ticket: &str) -> Result<(Hash, BlobFormat, EndpointAddr), String> {
let ticket: iroh_blobs::ticket::BlobTicket =
ticket.parse().map_err(|e| format!("invalid ticket: {e}"))?;
Ok((ticket.hash(), ticket.format(), ticket.node_addr().clone()))
let (addr, hash, format) = ticket.into_parts();
Ok((hash, format, addr))
}
/// Load the endpoint secret key from `path`, creating it (mode 0600) on
@@ -310,7 +255,7 @@ fn load_or_create_secret(path: &Path) -> Result<SecretKey> {
Ok(hex) => parse_secret(hex.trim())
.with_context(|| format!("parsing secret key {}", path.display())),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
let secret = SecretKey::generate(rand::rngs::OsRng);
let secret = SecretKey::generate();
let hex = hex_encode(&secret.to_bytes());
use std::io::Write;
use std::os::unix::fs::OpenOptionsExt;