Migrate to iroh 1.0 and iroh-blobs 0.103
The 0.35 pin's rationale ("the post-0.35 rewrite is not yet production
quality") expired when iroh hit 1.0 in June 2026: the rewrite line
(0.103) is now the production line and the only one receiving fixes.
The rewrite replaced wrappable store traits with an irpc-based API, so
the seams moved:
- shaped.rs: the 440-line ShapedStore trait wrapper becomes a provider
event handler. Trust gating (untrusted peers see only openly-served
hashes) now intercepts requests before any bytes move; upload rate
limiting rides the provider's Throttle hook; upload progress events
come from per-request update streams. The TokenBucket is unchanged.
- store.rs: FsStore's API handle replaces the store traits, and the
LocalPool machinery for non-Send futures is gone. GC is now the
store's built-in periodic mark-and-sweep, fed by a pin-roots snapshot
via the protect callback (new config knob gc_interval_secs, default
300); the on-demand Gc request answers Unimplemented, and the gc
conformance test polls the sweep instead.
- transfer.rs: BlobsProtocol + Router replace handle_connection, the
new multi-provider Downloader replaces the old queue, and mdns
discovery moved to the iroh-mdns-address-lookup crate (it left iroh
core in 1.0). Ticket-embedded provider addresses feed a MemoryLookup
address book. Endpoint presets: Minimal (LAN-only default) or N0
(wan_upload), preserving the old relay posture.
- Node* became Endpoint* throughout; announcement signatures use iroh's
own Signature type (ed25519-dalek dep dropped); iroh-io dropped.
Known regression: max_download_bytes_per_sec is currently not enforced
— download shaping rode the old store's batch writer and 0.103's
downloader has no equivalent seam yet.
Announcement wire format note: EndpointAddr serializes differently than
NodeAddr, so pre- and post-migration daemons won't parse each other's
LAN announcements. Announcements are live-only, nothing stored breaks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Generated
+1227
-1456
File diff suppressed because it is too large
Load Diff
+4
-5
@@ -12,11 +12,10 @@ rust-version = "1.85"
|
|||||||
[workspace.dependencies]
|
[workspace.dependencies]
|
||||||
varde-proto = { path = "varde-proto" }
|
varde-proto = { path = "varde-proto" }
|
||||||
|
|
||||||
# Pinned per spec: 0.35 is the recommended production line of iroh-blobs;
|
# iroh 1.0 (June 2026) and the post-rewrite iroh-blobs line, which is
|
||||||
# the post-0.35 rewrite is not yet production quality.
|
# now the production line (the 0.35 pin's rationale expired with 1.0).
|
||||||
iroh-blobs = "=0.35.0"
|
iroh-blobs = "0.103"
|
||||||
iroh = "0.35"
|
iroh = { version = "1", features = ["tls-ring"] }
|
||||||
iroh-io = "0.6"
|
|
||||||
reflink-copy = "0.1"
|
reflink-copy = "0.1"
|
||||||
|
|
||||||
serde = { version = "1", features = ["derive"] }
|
serde = { version = "1", features = ["derive"] }
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
A small, boring, distro-packageable Linux daemon that owns a
|
A small, boring, distro-packageable Linux daemon that owns a
|
||||||
content-addressed blob store and mirrors content between consenting
|
content-addressed blob store and mirrors content between consenting
|
||||||
peers, built on [iroh](https://iroh.computer) (iroh-blobs 0.35).
|
peers, built on [iroh](https://iroh.computer) (iroh 1.0, iroh-blobs 0.103).
|
||||||
Applications talk to it over a unix socket: "add this path", "pin this
|
Applications talk to it over a unix socket: "add this path", "pin this
|
||||||
hash", "materialize hash X at path Y". Think: what Windows Delivery
|
hash", "materialize hash X at path Y". Think: what Windows Delivery
|
||||||
Optimization is for updates — generalized, open, legible.
|
Optimization is for updates — generalized, open, legible.
|
||||||
|
|||||||
@@ -27,8 +27,8 @@ varde/
|
|||||||
```
|
```
|
||||||
|
|
||||||
**Key dependencies and versions:**
|
**Key dependencies and versions:**
|
||||||
- `iroh-blobs = "0.35"` — pin this. Per upstream, the post-0.35 rewrite is not yet production quality; 0.35 is the recommended production line. Use its persistent fs store (`iroh_blobs::store::fs`).
|
- `iroh-blobs = "0.103"` — the post-rewrite line, now the production line (the old 0.35 pin predated iroh 1.0). Use its persistent fs store (`iroh_blobs::store::fs`).
|
||||||
- `iroh` — matching version compatible with iroh-blobs 0.35.
|
- `iroh = "1"` — iroh 1.0, stable protocol and API.
|
||||||
- `tokio`, `serde`, `serde_json`, `tracing`, `clap` (ctl only), `anyhow`/`thiserror`.
|
- `tokio`, `serde`, `serde_json`, `tracing`, `clap` (ctl only), `anyhow`/`thiserror`.
|
||||||
- `zbus` for NetworkManager metered-status (feature-gated, `default-features` on Linux only).
|
- `zbus` for NetworkManager metered-status (feature-gated, `default-features` on Linux only).
|
||||||
|
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ fn ctl_status_round_trips() {
|
|||||||
max_download_bytes_per_sec: 0,
|
max_download_bytes_per_sec: 0,
|
||||||
discovery: false,
|
discovery: false,
|
||||||
wan_upload: false,
|
wan_upload: false,
|
||||||
|
gc_interval_secs: 300,
|
||||||
};
|
};
|
||||||
|
|
||||||
let rt = tokio::runtime::Builder::new_multi_thread()
|
let rt = tokio::runtime::Builder::new_multi_thread()
|
||||||
|
|||||||
+8
-10
@@ -14,22 +14,20 @@ required-features = ["socket-api"]
|
|||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
varde-proto = { workspace = true }
|
varde-proto = { workspace = true }
|
||||||
iroh = { workspace = true, features = ["discovery-local-network"] }
|
iroh = { workspace = true }
|
||||||
iroh-blobs = { workspace = true }
|
iroh-blobs = { workspace = true }
|
||||||
rand = "0.8"
|
# mDNS/local-network address lookup left iroh core in 1.0.
|
||||||
# The next three exist to interoperate with iroh-blobs trait signatures
|
iroh-mdns-address-lookup = "0.4"
|
||||||
# and channels; all are already in the dependency tree via iroh.
|
# Interop with iroh-blobs streams and channels; both are already in the
|
||||||
|
# dependency tree via iroh-blobs.
|
||||||
n0-future = "0.1"
|
n0-future = "0.1"
|
||||||
async-channel = "2"
|
irpc = "0.17"
|
||||||
futures-lite = "2"
|
|
||||||
bytes = "1"
|
bytes = "1"
|
||||||
# Feature-gated D-Bus client for NetworkManager metered status.
|
# Feature-gated D-Bus client for NetworkManager metered status.
|
||||||
zbus = { version = "5", optional = true, default-features = false, features = ["tokio"] }
|
zbus = { version = "5", optional = true, default-features = false, features = ["tokio"] }
|
||||||
# Announcement signatures: same ed25519 implementation iroh keys use,
|
# Announcement signatures use iroh's own key types; postcard is the
|
||||||
# postcard is the deterministic encoding signed over.
|
# deterministic encoding signed over.
|
||||||
ed25519-dalek = "2"
|
|
||||||
postcard = { version = "1", features = ["use-std"] }
|
postcard = { version = "1", features = ["use-std"] }
|
||||||
iroh-io = { workspace = true }
|
|
||||||
reflink-copy = { workspace = true }
|
reflink-copy = { workspace = true }
|
||||||
anyhow = { workspace = true }
|
anyhow = { workspace = true }
|
||||||
serde = { workspace = true }
|
serde = { workspace = true }
|
||||||
|
|||||||
@@ -24,6 +24,10 @@ pub struct Config {
|
|||||||
/// Whether any WAN (non-link-local) upload is permitted. Default off:
|
/// Whether any WAN (non-link-local) upload is permitted. Default off:
|
||||||
/// LAN-only posture with zero WAN upload.
|
/// LAN-only posture with zero WAN upload.
|
||||||
pub wan_upload: bool,
|
pub wan_upload: bool,
|
||||||
|
/// Interval of the store's built-in garbage collector in seconds.
|
||||||
|
/// Since iroh-blobs 0.103 there is no on-demand gc; unpinned blobs
|
||||||
|
/// are swept by this loop.
|
||||||
|
pub gc_interval_secs: u64,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Serde image of the TOML config file. Everything optional so an empty
|
/// Serde image of the TOML config file. Everything optional so an empty
|
||||||
@@ -37,6 +41,7 @@ struct FileConfig {
|
|||||||
max_download_bytes_per_sec: Option<u64>,
|
max_download_bytes_per_sec: Option<u64>,
|
||||||
discovery: Option<bool>,
|
discovery: Option<bool>,
|
||||||
wan_upload: Option<bool>,
|
wan_upload: Option<bool>,
|
||||||
|
gc_interval_secs: Option<u64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Values collected from CLI flags; `None` means "not given".
|
/// Values collected from CLI flags; `None` means "not given".
|
||||||
@@ -171,6 +176,9 @@ impl Config {
|
|||||||
wan_upload: env_bool("VARDE_WAN_UPLOAD")?
|
wan_upload: env_bool("VARDE_WAN_UPLOAD")?
|
||||||
.or(file.wan_upload)
|
.or(file.wan_upload)
|
||||||
.unwrap_or(false),
|
.unwrap_or(false),
|
||||||
|
gc_interval_secs: env_u64("VARDE_GC_INTERVAL")?
|
||||||
|
.or(file.gc_interval_secs)
|
||||||
|
.unwrap_or(300),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+40
-26
@@ -51,8 +51,30 @@ impl Daemon {
|
|||||||
pub async fn open(config: Config) -> anyhow::Result<Arc<Daemon>> {
|
pub async fn open(config: Config) -> anyhow::Result<Arc<Daemon>> {
|
||||||
std::fs::create_dir_all(&config.store_dir)
|
std::fs::create_dir_all(&config.store_dir)
|
||||||
.with_context(|| format!("creating store directory {}", config.store_dir.display()))?;
|
.with_context(|| format!("creating store directory {}", config.store_dir.display()))?;
|
||||||
let store = BlobStore::open(&config.store_dir).await?;
|
|
||||||
let meta = Arc::new(Meta::load(&config.store_dir)?);
|
let meta = Arc::new(Meta::load(&config.store_dir)?);
|
||||||
|
// The store's built-in gc keeps whatever this snapshot reports
|
||||||
|
// (plus tags and in-flight temp tags) on every run.
|
||||||
|
let gc_roots: crate::store::GcRootsFn = {
|
||||||
|
let meta = meta.clone();
|
||||||
|
Arc::new(move || {
|
||||||
|
meta.pins()
|
||||||
|
.into_iter()
|
||||||
|
.filter_map(|(hex, _)| {
|
||||||
|
let hash = Hash::from_str(&hex).ok()?;
|
||||||
|
Some(HashAndFormat {
|
||||||
|
hash,
|
||||||
|
format: blob_format(meta.format_of(&hex)),
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let store = BlobStore::open(
|
||||||
|
&config.store_dir,
|
||||||
|
Duration::from_secs(config.gc_interval_secs.max(1)),
|
||||||
|
gc_roots,
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
// Capacity bounds memory if a subscriber stalls; laggards get a
|
// Capacity bounds memory if a subscriber stalls; laggards get a
|
||||||
// Lagged error, not unbounded buffering.
|
// Lagged error, not unbounded buffering.
|
||||||
let (events, _) = broadcast::channel(1024);
|
let (events, _) = broadcast::channel(1024);
|
||||||
@@ -68,7 +90,6 @@ impl Daemon {
|
|||||||
&store,
|
&store,
|
||||||
meta.clone(),
|
meta.clone(),
|
||||||
open_filter,
|
open_filter,
|
||||||
store.pool_handle(),
|
|
||||||
events.clone(),
|
events.clone(),
|
||||||
metered,
|
metered,
|
||||||
)
|
)
|
||||||
@@ -194,7 +215,7 @@ impl Daemon {
|
|||||||
|
|
||||||
/// Track presence for a discovery sighting of `node_id`. Content
|
/// Track presence for a discovery sighting of `node_id`. Content
|
||||||
/// sync happens via the announcement path, not here.
|
/// sync happens via the announcement path, not here.
|
||||||
fn on_peer_seen(&self, node_id: iroh::NodeId) {
|
fn on_peer_seen(&self, node_id: iroh::EndpointId) {
|
||||||
let id = node_id.to_string();
|
let id = node_id.to_string();
|
||||||
let rejoined = {
|
let rejoined = {
|
||||||
let mut peers = self.lan_peers.lock().expect("lan peers lock");
|
let mut peers = self.lan_peers.lock().expect("lan peers lock");
|
||||||
@@ -247,10 +268,10 @@ impl Daemon {
|
|||||||
if let Ok((_, _, true)) = self.store.presence(root).await {
|
if let Ok((_, _, true)) = self.store.presence(root).await {
|
||||||
return; // already complete
|
return; // already complete
|
||||||
}
|
}
|
||||||
let providers: Vec<iroh::NodeAddr> = ann
|
let providers: Vec<iroh::EndpointAddr> = ann
|
||||||
.providers
|
.providers
|
||||||
.iter()
|
.iter()
|
||||||
.filter(|p| self.meta.is_trusted(&p.node_id.to_string()))
|
.filter(|p| self.meta.is_trusted(&p.id.to_string()))
|
||||||
.cloned()
|
.cloned()
|
||||||
.collect();
|
.collect();
|
||||||
if providers.is_empty() {
|
if providers.is_empty() {
|
||||||
@@ -269,7 +290,7 @@ impl Daemon {
|
|||||||
/// Queue fetches for every incomplete pin from a trusted LAN peer.
|
/// Queue fetches for every incomplete pin from a trusted LAN peer.
|
||||||
/// The downloader dedupes by content, so repeated sightings are
|
/// The downloader dedupes by content, so repeated sightings are
|
||||||
/// cheap; discovery resolves the peer's addresses.
|
/// cheap; discovery resolves the peer's addresses.
|
||||||
async fn sync_pins_from(&self, node_id: iroh::NodeId) {
|
async fn sync_pins_from(&self, node_id: iroh::EndpointId) {
|
||||||
for (hex, _record) in self.meta.pins() {
|
for (hex, _record) in self.meta.pins() {
|
||||||
let Ok(hash) = Hash::from_str(&hex) else {
|
let Ok(hash) = Hash::from_str(&hex) else {
|
||||||
continue;
|
continue;
|
||||||
@@ -283,7 +304,7 @@ impl Daemon {
|
|||||||
hash,
|
hash,
|
||||||
format: blob_format(self.meta.format_of(&hex)),
|
format: blob_format(self.meta.format_of(&hex)),
|
||||||
},
|
},
|
||||||
vec![iroh::NodeAddr::new(node_id)],
|
vec![iroh::EndpointAddr::from(node_id)],
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
Err(e) => warn!(hash = %hex, error = %e, "checking pin presence"),
|
Err(e) => warn!(hash = %hex, error = %e, "checking pin presence"),
|
||||||
@@ -334,11 +355,11 @@ impl Daemon {
|
|||||||
// Fetch if absent: try every trusted peer currently
|
// Fetch if absent: try every trusted peer currently
|
||||||
// present on the LAN.
|
// present on the LAN.
|
||||||
if let Ok((_, _, false)) = self.store.presence(parsed).await {
|
if let Ok((_, _, false)) = self.store.presence(parsed).await {
|
||||||
let present: Vec<iroh::NodeAddr> = self
|
let present: Vec<iroh::EndpointAddr> = self
|
||||||
.present_trusted_peers()
|
.present_trusted_peers()
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter(|p| p.connected)
|
.filter(|p| p.connected)
|
||||||
.filter_map(|p| p.node_id.parse().ok().map(iroh::NodeAddr::new))
|
.filter_map(|p| p.node_id.parse::<iroh::EndpointId>().ok().map(iroh::EndpointAddr::from))
|
||||||
.collect();
|
.collect();
|
||||||
if !present.is_empty() {
|
if !present.is_empty() {
|
||||||
self.transfer.spawn_fetch(
|
self.transfer.spawn_fetch(
|
||||||
@@ -414,9 +435,14 @@ impl Daemon {
|
|||||||
Ok(ResponseData::Pins { pins })
|
Ok(ResponseData::Pins { pins })
|
||||||
}
|
}
|
||||||
Request::Gc {} => {
|
Request::Gc {} => {
|
||||||
let roots = self.pin_roots()?;
|
// iroh-blobs 0.103 removed on-demand gc: the store runs
|
||||||
let blobs_removed = self.store.gc(roots).await?;
|
// its own periodic mark-and-sweep, protecting pins via
|
||||||
Ok(ResponseData::GcDone { blobs_removed })
|
// the roots snapshot handed to BlobStore::open.
|
||||||
|
Err(OpError::Unimplemented(
|
||||||
|
"gc now runs continuously inside the blob store; \
|
||||||
|
on-demand gc has no API in iroh-blobs 0.103"
|
||||||
|
.to_string(),
|
||||||
|
))
|
||||||
}
|
}
|
||||||
Request::TicketExport { hash } => {
|
Request::TicketExport { hash } => {
|
||||||
let parsed = parse_hash(&hash)?;
|
let parsed = parse_hash(&hash)?;
|
||||||
@@ -458,7 +484,7 @@ impl Daemon {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
Request::PeerTrust { node_id } => {
|
Request::PeerTrust { node_id } => {
|
||||||
let parsed: iroh::NodeId = node_id
|
let parsed: iroh::EndpointId = node_id
|
||||||
.parse()
|
.parse()
|
||||||
.map_err(|e| OpError::InvalidArgument(format!("invalid node id: {e}")))?;
|
.map_err(|e| OpError::InvalidArgument(format!("invalid node id: {e}")))?;
|
||||||
self.meta.trust_peer(&parsed.to_string())?;
|
self.meta.trust_peer(&parsed.to_string())?;
|
||||||
@@ -476,7 +502,7 @@ impl Daemon {
|
|||||||
Ok(ResponseData::Done {})
|
Ok(ResponseData::Done {})
|
||||||
}
|
}
|
||||||
Request::PeerUntrust { node_id } => {
|
Request::PeerUntrust { node_id } => {
|
||||||
let parsed: iroh::NodeId = node_id
|
let parsed: iroh::EndpointId = node_id
|
||||||
.parse()
|
.parse()
|
||||||
.map_err(|e| OpError::InvalidArgument(format!("invalid node id: {e}")))?;
|
.map_err(|e| OpError::InvalidArgument(format!("invalid node id: {e}")))?;
|
||||||
if !self.meta.untrust_peer(&parsed.to_string())? {
|
if !self.meta.untrust_peer(&parsed.to_string())? {
|
||||||
@@ -491,18 +517,6 @@ impl Daemon {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The GC roots: every pinned hash with its recorded format.
|
|
||||||
fn pin_roots(&self) -> Result<Vec<HashAndFormat>, OpError> {
|
|
||||||
let mut roots = Vec::new();
|
|
||||||
for (hash, _record) in self.meta.pins() {
|
|
||||||
let parsed = parse_hash(&hash)?;
|
|
||||||
roots.push(HashAndFormat {
|
|
||||||
hash: parsed,
|
|
||||||
format: blob_format(self.meta.format_of(&hash)),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
Ok(roots)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn blob_format(format: StoredFormat) -> BlobFormat {
|
fn blob_format(format: StoredFormat) -> BlobFormat {
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ use std::pin::Pin;
|
|||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
use anyhow::Result;
|
use anyhow::Result;
|
||||||
use iroh::{NodeAddr, NodeId, PublicKey, SecretKey};
|
use iroh::{EndpointAddr, EndpointId, PublicKey, SecretKey};
|
||||||
use iroh_blobs::Hash;
|
use iroh_blobs::Hash;
|
||||||
use n0_future::Stream;
|
use n0_future::Stream;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
@@ -47,7 +47,7 @@ pub struct Announcement {
|
|||||||
/// Free-form metadata (name hints, sizes, provenance...).
|
/// Free-form metadata (name hints, sizes, provenance...).
|
||||||
pub meta: BTreeMap<String, String>,
|
pub meta: BTreeMap<String, String>,
|
||||||
/// Peers believed to hold the content.
|
/// Peers believed to hold the content.
|
||||||
pub providers: Vec<NodeAddr>,
|
pub providers: Vec<EndpointAddr>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// An [`Announcement`] plus the author's signature over it and its topic.
|
/// An [`Announcement`] plus the author's signature over it and its topic.
|
||||||
@@ -83,7 +83,7 @@ impl SignedAnnouncement {
|
|||||||
.as_slice()
|
.as_slice()
|
||||||
.try_into()
|
.try_into()
|
||||||
.map_err(|_| anyhow::anyhow!("signature must be 64 bytes"))?;
|
.map_err(|_| anyhow::anyhow!("signature must be 64 bytes"))?;
|
||||||
let signature = ed25519_dalek::Signature::from_bytes(bytes);
|
let signature = iroh::Signature::from_bytes(bytes);
|
||||||
self.announcement
|
self.announcement
|
||||||
.author
|
.author
|
||||||
.verify(&payload, &signature)
|
.verify(&payload, &signature)
|
||||||
@@ -96,7 +96,7 @@ impl SignedAnnouncement {
|
|||||||
/// are excluded — they are hints that relays may legitimately refresh —
|
/// are excluded — they are hints that relays may legitimately refresh —
|
||||||
/// but the provider identities are covered.
|
/// but the provider identities are covered.
|
||||||
fn signing_payload(topic: TopicKey, announcement: &Announcement) -> Result<Vec<u8>> {
|
fn signing_payload(topic: TopicKey, announcement: &Announcement) -> Result<Vec<u8>> {
|
||||||
let provider_ids: Vec<NodeId> = announcement.providers.iter().map(|p| p.node_id).collect();
|
let provider_ids: Vec<EndpointId> = announcement.providers.iter().map(|p| p.id).collect();
|
||||||
let payload = (
|
let payload = (
|
||||||
topic,
|
topic,
|
||||||
announcement.root,
|
announcement.root,
|
||||||
@@ -127,7 +127,7 @@ pub trait DiscoveryProvider: Send + Sync {
|
|||||||
/// broadcasts its content list.
|
/// broadcasts its content list.
|
||||||
pub struct LanDiscovery {
|
pub struct LanDiscovery {
|
||||||
secret: SecretKey,
|
secret: SecretKey,
|
||||||
sightings: std::sync::Mutex<Option<mpsc::Receiver<NodeId>>>,
|
sightings: std::sync::Mutex<Option<mpsc::Receiver<EndpointId>>>,
|
||||||
/// Snapshot of locally pinned roots, queried per sighting.
|
/// Snapshot of locally pinned roots, queried per sighting.
|
||||||
pins: Arc<dyn Fn() -> Vec<Hash> + Send + Sync>,
|
pins: Arc<dyn Fn() -> Vec<Hash> + Send + Sync>,
|
||||||
}
|
}
|
||||||
@@ -135,7 +135,7 @@ pub struct LanDiscovery {
|
|||||||
impl LanDiscovery {
|
impl LanDiscovery {
|
||||||
pub fn new(
|
pub fn new(
|
||||||
secret: SecretKey,
|
secret: SecretKey,
|
||||||
sightings: mpsc::Receiver<NodeId>,
|
sightings: mpsc::Receiver<EndpointId>,
|
||||||
pins: Arc<dyn Fn() -> Vec<Hash> + Send + Sync>,
|
pins: Arc<dyn Fn() -> Vec<Hash> + Send + Sync>,
|
||||||
) -> LanDiscovery {
|
) -> LanDiscovery {
|
||||||
LanDiscovery {
|
LanDiscovery {
|
||||||
@@ -165,7 +165,7 @@ impl DiscoveryProvider for LanDiscovery {
|
|||||||
root,
|
root,
|
||||||
author: secret.public(),
|
author: secret.public(),
|
||||||
meta: BTreeMap::from([("source".to_string(), "mdns".to_string())]),
|
meta: BTreeMap::from([("source".to_string(), "mdns".to_string())]),
|
||||||
providers: vec![NodeAddr::new(node_id)],
|
providers: vec![EndpointAddr::new(node_id)],
|
||||||
};
|
};
|
||||||
match SignedAnnouncement::sign(&secret, LAN_TOPIC, announcement) {
|
match SignedAnnouncement::sign(&secret, LAN_TOPIC, announcement) {
|
||||||
Ok(signed) => {
|
Ok(signed) => {
|
||||||
@@ -197,7 +197,7 @@ mod tests {
|
|||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
fn keypair() -> SecretKey {
|
fn keypair() -> SecretKey {
|
||||||
SecretKey::generate(rand::rngs::OsRng)
|
SecretKey::generate()
|
||||||
}
|
}
|
||||||
|
|
||||||
fn sample(author: PublicKey) -> Announcement {
|
fn sample(author: PublicKey) -> Announcement {
|
||||||
@@ -205,7 +205,7 @@ mod tests {
|
|||||||
root: Hash::new(b"content"),
|
root: Hash::new(b"content"),
|
||||||
author,
|
author,
|
||||||
meta: BTreeMap::from([("name".to_string(), "demo".to_string())]),
|
meta: BTreeMap::from([("name".to_string(), "demo".to_string())]),
|
||||||
providers: vec![NodeAddr::new(keypair().public())],
|
providers: vec![EndpointAddr::new(keypair().public())],
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+164
-332
@@ -1,29 +1,31 @@
|
|||||||
//! Traffic shaping and per-hash gating for the blob store.
|
//! Traffic shaping and per-hash gating for the blob provider.
|
||||||
//!
|
//!
|
||||||
//! [`ShapedStore`] wraps the iroh-blobs fs store and implements the full
|
//! iroh-blobs 0.103 replaced wrappable store traits with a provider
|
||||||
//! `Store` trait, adding two token buckets: reads of entry data (the
|
//! event system, so shaping moved from the store to the event seam:
|
||||||
//! bytes a provider sends to peers) draw from the upload bucket, batch
|
//! [`provider_events`] spawns a handler that intercepts incoming
|
||||||
//! writes of downloaded data draw from the download bucket. An optional
|
//! connections and get requests (trust gating: untrusted peers may only
|
||||||
//! filter hides hashes from `get`, which is how untrusted peers are
|
//! fetch openly-served hashes), throttles upload bytes through a token
|
||||||
//! limited to openly-served content — the provider simply answers "not
|
//! bucket, and forwards transfer progress into the daemon event stream.
|
||||||
//! found" for everything else.
|
//!
|
||||||
|
//! Download rate limiting rode on the old store's batch writer and has
|
||||||
|
//! no equivalent seam in 0.103's downloader; `max_download_bytes_per_sec`
|
||||||
|
//! is currently not enforced (see the migration commit).
|
||||||
|
|
||||||
use std::future::Future;
|
use std::collections::HashMap;
|
||||||
use std::io;
|
|
||||||
use std::path::PathBuf;
|
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
use iroh_blobs::store::bao_tree::io::fsm::{BaoContentItem, Outboard};
|
use iroh_blobs::provider::events::{
|
||||||
use iroh_blobs::store::fs::Store as FsStore;
|
AbortReason, ClientConnected, ConnectMode, EventMask, EventSender, ObserveMode,
|
||||||
use iroh_blobs::store::{
|
ProviderMessage, RequestMode, ThrottleMode,
|
||||||
BaoBatchWriter, ConsistencyCheckProgress, DbIter, EntryStatus, ExportMode, ExportProgressCb,
|
|
||||||
ImportMode, ImportProgress, Map, MapEntry, MapEntryMut, MapMut, ReadableStore, Store,
|
|
||||||
};
|
};
|
||||||
use iroh_blobs::util::progress::{BoxedProgressSender, IdGenerator, ProgressSender};
|
use iroh_blobs::Hash;
|
||||||
use iroh_blobs::util::Tag;
|
use tokio::sync::broadcast;
|
||||||
use iroh_blobs::{BlobFormat, Hash, HashAndFormat, TempTag};
|
use tracing::debug;
|
||||||
use iroh_io::AsyncSliceReader;
|
use varde_proto::{Direction, Event};
|
||||||
|
|
||||||
|
use crate::meta::Meta;
|
||||||
|
use crate::metered::MeteredState;
|
||||||
|
|
||||||
/// An async token bucket. Rate 0 means unlimited.
|
/// An async token bucket. Rate 0 means unlimited.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
@@ -81,332 +83,162 @@ impl TokenBucket {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Predicate deciding whether a hash may be served on this store view.
|
/// Predicate deciding whether a hash may be served to untrusted peers.
|
||||||
pub type ServeFilter = Arc<dyn Fn(&Hash) -> bool + Send + Sync>;
|
pub type ServeFilter = Arc<dyn Fn(&Hash) -> bool + Send + Sync>;
|
||||||
|
|
||||||
/// The fs store wrapped with rate limiting and an optional serve filter.
|
/// Provider gate configuration.
|
||||||
#[derive(Clone)]
|
pub struct Gate {
|
||||||
pub struct ShapedStore {
|
pub meta: Arc<Meta>,
|
||||||
inner: FsStore,
|
pub open_filter: ServeFilter,
|
||||||
upload: Arc<TokenBucket>,
|
pub upload: Arc<TokenBucket>,
|
||||||
download: Arc<TokenBucket>,
|
pub serving_enabled: bool,
|
||||||
filter: Option<ServeFilter>,
|
pub metered: MeteredState,
|
||||||
|
pub events: broadcast::Sender<Event>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::fmt::Debug for ShapedStore {
|
/// Spawn the provider event handler and return the [`EventSender`] to
|
||||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
/// wire into [`iroh_blobs::BlobsProtocol`].
|
||||||
f.debug_struct("ShapedStore")
|
pub fn provider_events(gate: Gate) -> EventSender {
|
||||||
.field("inner", &self.inner)
|
let mask = EventMask {
|
||||||
.field("upload", &self.upload)
|
connected: ConnectMode::Intercept,
|
||||||
.field("download", &self.download)
|
get: RequestMode::InterceptLog,
|
||||||
.field("filtered", &self.filter.is_some())
|
get_many: RequestMode::InterceptLog,
|
||||||
.finish()
|
push: RequestMode::Disabled,
|
||||||
|
observe: ObserveMode::Intercept,
|
||||||
|
throttle: ThrottleMode::Intercept,
|
||||||
|
};
|
||||||
|
let (sender, mut rx) = EventSender::channel(32, mask);
|
||||||
|
tokio::spawn(async move {
|
||||||
|
// Trust decided once per connection, consulted per request.
|
||||||
|
let mut trusted_conns: HashMap<u64, bool> = HashMap::new();
|
||||||
|
while let Some(msg) = rx.recv().await {
|
||||||
|
match msg {
|
||||||
|
ProviderMessage::ClientConnected(m) => {
|
||||||
|
let allowed = gate.serving_enabled && !gate.metered.is_metered();
|
||||||
|
let trusted = m
|
||||||
|
.inner
|
||||||
|
.endpoint_id
|
||||||
|
.map(|id| gate.meta.is_trusted(&id.to_string()))
|
||||||
|
.unwrap_or(false);
|
||||||
|
let ClientConnected {
|
||||||
|
connection_id,
|
||||||
|
endpoint_id,
|
||||||
|
} = m.inner;
|
||||||
|
debug!(?endpoint_id, trusted, allowed, "incoming blobs connection");
|
||||||
|
if allowed {
|
||||||
|
trusted_conns.insert(connection_id, trusted);
|
||||||
}
|
}
|
||||||
|
let verdict = if allowed {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(AbortReason::Permission)
|
||||||
|
};
|
||||||
|
m.tx.send(verdict).await.ok();
|
||||||
|
}
|
||||||
|
ProviderMessage::ConnectionClosed(m) => {
|
||||||
|
trusted_conns.remove(&m.inner.connection_id);
|
||||||
|
}
|
||||||
|
ProviderMessage::GetRequestReceived(m) => {
|
||||||
|
let allowed = allow(&gate, &trusted_conns, m.inner.connection_id, [
|
||||||
|
m.inner.request.hash,
|
||||||
|
]);
|
||||||
|
let verdict = if allowed {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(AbortReason::Permission)
|
||||||
|
};
|
||||||
|
m.tx.send(verdict).await.ok();
|
||||||
|
if allowed {
|
||||||
|
forward_progress(m.rx, gate.events.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ProviderMessage::GetManyRequestReceived(m) => {
|
||||||
|
let allowed = allow(
|
||||||
|
&gate,
|
||||||
|
&trusted_conns,
|
||||||
|
m.inner.connection_id,
|
||||||
|
m.inner.request.hashes.iter().copied(),
|
||||||
|
);
|
||||||
|
let verdict = if allowed {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(AbortReason::Permission)
|
||||||
|
};
|
||||||
|
m.tx.send(verdict).await.ok();
|
||||||
|
if allowed {
|
||||||
|
forward_progress(m.rx, gate.events.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ProviderMessage::ObserveRequestReceived(m) => {
|
||||||
|
// Presence is information too: gate it like data.
|
||||||
|
let allowed = allow(&gate, &trusted_conns, m.inner.connection_id, [
|
||||||
|
m.inner.request.hash,
|
||||||
|
]);
|
||||||
|
let verdict = if allowed {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(AbortReason::Permission)
|
||||||
|
};
|
||||||
|
m.tx.send(verdict).await.ok();
|
||||||
|
}
|
||||||
|
ProviderMessage::Throttle(m) => {
|
||||||
|
// Pay for the chunk from the shared upload bucket off
|
||||||
|
// the handler loop, so a deep debt doesn't stall
|
||||||
|
// unrelated connections' gating decisions.
|
||||||
|
let bucket = gate.upload.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
bucket.acquire(m.inner.size as usize).await;
|
||||||
|
m.tx.send(Ok(())).await.ok();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
// Push is disabled in the mask; notify variants are not
|
||||||
|
// subscribed. Ignore anything else.
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
sender
|
||||||
}
|
}
|
||||||
|
|
||||||
impl ShapedStore {
|
/// A request is allowed if the connection is trusted, or every requested
|
||||||
pub fn new(
|
/// hash is openly served.
|
||||||
inner: FsStore,
|
fn allow(
|
||||||
upload: Arc<TokenBucket>,
|
gate: &Gate,
|
||||||
download: Arc<TokenBucket>,
|
trusted_conns: &HashMap<u64, bool>,
|
||||||
) -> ShapedStore {
|
connection_id: u64,
|
||||||
ShapedStore {
|
hashes: impl IntoIterator<Item = Hash>,
|
||||||
inner,
|
) -> bool {
|
||||||
upload,
|
if trusted_conns.get(&connection_id).copied().unwrap_or(false) {
|
||||||
download,
|
return true;
|
||||||
filter: None,
|
|
||||||
}
|
}
|
||||||
|
hashes.into_iter().all(|hash| (gate.open_filter)(&hash))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A view of the same store that only serves hashes passing `filter`.
|
/// Forward per-request transfer updates into the daemon event stream.
|
||||||
pub fn filtered(&self, filter: ServeFilter) -> ShapedStore {
|
fn forward_progress(
|
||||||
ShapedStore {
|
mut rx: irpc::channel::mpsc::Receiver<iroh_blobs::provider::events::RequestUpdate>,
|
||||||
inner: self.inner.clone(),
|
events: broadcast::Sender<Event>,
|
||||||
upload: self.upload.clone(),
|
) {
|
||||||
download: self.download.clone(),
|
use iroh_blobs::provider::events::RequestUpdate;
|
||||||
filter: Some(filter),
|
tokio::spawn(async move {
|
||||||
|
let mut current: Option<(Hash, u64)> = None;
|
||||||
|
while let Ok(Some(update)) = rx.recv().await {
|
||||||
|
match update {
|
||||||
|
RequestUpdate::Started(s) => current = Some((s.hash, s.size)),
|
||||||
|
RequestUpdate::Progress(p) => {
|
||||||
|
if let Some((hash, size)) = ¤t {
|
||||||
|
let _ = events.send(Event::TransferProgress {
|
||||||
|
hash: hash.to_hex().to_string(),
|
||||||
|
direction: Direction::Upload,
|
||||||
|
bytes_done: p.end_offset,
|
||||||
|
bytes_total: Some(*size),
|
||||||
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
RequestUpdate::Completed(_) | RequestUpdate::Aborted(_) => break,
|
||||||
|
|
||||||
/// An entry whose data reads draw from the upload bucket.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
pub struct ShapedEntry {
|
|
||||||
inner: <FsStore as Map>::Entry,
|
|
||||||
upload: Arc<TokenBucket>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl MapEntry for ShapedEntry {
|
|
||||||
fn hash(&self) -> Hash {
|
|
||||||
self.inner.hash()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn size(&self) -> iroh_blobs::store::BaoBlobSize {
|
|
||||||
self.inner.size()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn is_complete(&self) -> bool {
|
|
||||||
self.inner.is_complete()
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn outboard(&self) -> io::Result<impl Outboard> {
|
|
||||||
// The fs entry's inherent (synchronous) methods shadow the trait
|
|
||||||
// methods here and below.
|
|
||||||
self.inner.outboard()
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn data_reader(&self) -> io::Result<impl AsyncSliceReader> {
|
|
||||||
let inner = self.inner.data_reader();
|
|
||||||
Ok(ThrottledReader {
|
|
||||||
inner,
|
|
||||||
bucket: self.upload.clone(),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
});
|
||||||
/// AsyncSliceReader that pays for bytes read from the upload bucket.
|
|
||||||
#[derive(Debug)]
|
|
||||||
struct ThrottledReader<R> {
|
|
||||||
inner: R,
|
|
||||||
bucket: Arc<TokenBucket>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl<R: AsyncSliceReader> AsyncSliceReader for ThrottledReader<R> {
|
|
||||||
async fn read_at(&mut self, offset: u64, len: usize) -> io::Result<bytes::Bytes> {
|
|
||||||
let bytes = self.inner.read_at(offset, len).await?;
|
|
||||||
self.bucket.acquire(bytes.len()).await;
|
|
||||||
Ok(bytes)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn size(&mut self) -> io::Result<u64> {
|
|
||||||
self.inner.size().await
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Map for ShapedStore {
|
|
||||||
type Entry = ShapedEntry;
|
|
||||||
|
|
||||||
async fn get(&self, hash: &Hash) -> io::Result<Option<ShapedEntry>> {
|
|
||||||
if let Some(filter) = &self.filter {
|
|
||||||
if !filter(hash) {
|
|
||||||
return Ok(None);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(self.inner.get(hash).await?.map(|inner| ShapedEntry {
|
|
||||||
inner,
|
|
||||||
upload: self.upload.clone(),
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A mutable entry whose batch writes draw from the download bucket.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
pub struct ShapedEntryMut {
|
|
||||||
inner: <FsStore as MapMut>::EntryMut,
|
|
||||||
upload: Arc<TokenBucket>,
|
|
||||||
download: Arc<TokenBucket>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl MapEntry for ShapedEntryMut {
|
|
||||||
fn hash(&self) -> Hash {
|
|
||||||
self.inner.hash()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn size(&self) -> iroh_blobs::store::BaoBlobSize {
|
|
||||||
self.inner.size()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn is_complete(&self) -> bool {
|
|
||||||
self.inner.is_complete()
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn outboard(&self) -> io::Result<impl Outboard> {
|
|
||||||
// The fs entry's inherent (synchronous) methods shadow the trait
|
|
||||||
// methods here and below.
|
|
||||||
self.inner.outboard()
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn data_reader(&self) -> io::Result<impl AsyncSliceReader> {
|
|
||||||
let inner = self.inner.data_reader();
|
|
||||||
Ok(ThrottledReader {
|
|
||||||
inner,
|
|
||||||
bucket: self.upload.clone(),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl MapEntryMut for ShapedEntryMut {
|
|
||||||
async fn batch_writer(&self) -> io::Result<impl BaoBatchWriter> {
|
|
||||||
let inner = self.inner.batch_writer().await?;
|
|
||||||
Ok(ThrottledBatchWriter {
|
|
||||||
inner,
|
|
||||||
bucket: self.download.clone(),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// BaoBatchWriter that pays for leaf bytes written from the download
|
|
||||||
/// bucket.
|
|
||||||
struct ThrottledBatchWriter<W> {
|
|
||||||
inner: W,
|
|
||||||
bucket: Arc<TokenBucket>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl<W: BaoBatchWriter + Send> BaoBatchWriter for ThrottledBatchWriter<W> {
|
|
||||||
async fn write_batch(&mut self, size: u64, batch: Vec<BaoContentItem>) -> io::Result<()> {
|
|
||||||
let bytes: usize = batch
|
|
||||||
.iter()
|
|
||||||
.map(|item| match item {
|
|
||||||
BaoContentItem::Leaf(leaf) => leaf.data.len(),
|
|
||||||
// Parent nodes are two hashes of overhead.
|
|
||||||
BaoContentItem::Parent(_) => 64,
|
|
||||||
})
|
|
||||||
.sum();
|
|
||||||
self.bucket.acquire(bytes).await;
|
|
||||||
self.inner.write_batch(size, batch).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn sync(&mut self) -> io::Result<()> {
|
|
||||||
self.inner.sync().await
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl MapMut for ShapedStore {
|
|
||||||
type EntryMut = ShapedEntryMut;
|
|
||||||
|
|
||||||
async fn get_mut(&self, hash: &Hash) -> io::Result<Option<ShapedEntryMut>> {
|
|
||||||
Ok(self.inner.get_mut(hash).await?.map(|inner| ShapedEntryMut {
|
|
||||||
inner,
|
|
||||||
upload: self.upload.clone(),
|
|
||||||
download: self.download.clone(),
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn get_or_create(&self, hash: Hash, size: u64) -> io::Result<ShapedEntryMut> {
|
|
||||||
let inner = self.inner.get_or_create(hash, size).await?;
|
|
||||||
Ok(ShapedEntryMut {
|
|
||||||
inner,
|
|
||||||
upload: self.upload.clone(),
|
|
||||||
download: self.download.clone(),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn entry_status(&self, hash: &Hash) -> io::Result<EntryStatus> {
|
|
||||||
self.inner.entry_status(hash).await
|
|
||||||
}
|
|
||||||
|
|
||||||
fn entry_status_sync(&self, hash: &Hash) -> io::Result<EntryStatus> {
|
|
||||||
self.inner.entry_status_sync(hash)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn insert_complete(&self, entry: ShapedEntryMut) -> io::Result<()> {
|
|
||||||
self.inner.insert_complete(entry.inner).await
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl ReadableStore for ShapedStore {
|
|
||||||
async fn blobs(&self) -> io::Result<DbIter<Hash>> {
|
|
||||||
self.inner.blobs().await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn tags(
|
|
||||||
&self,
|
|
||||||
from: Option<Tag>,
|
|
||||||
to: Option<Tag>,
|
|
||||||
) -> io::Result<DbIter<(Tag, HashAndFormat)>> {
|
|
||||||
self.inner.tags(from, to).await
|
|
||||||
}
|
|
||||||
|
|
||||||
fn temp_tags(&self) -> Box<dyn Iterator<Item = HashAndFormat> + Send + Sync + 'static> {
|
|
||||||
self.inner.temp_tags()
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn consistency_check(
|
|
||||||
&self,
|
|
||||||
repair: bool,
|
|
||||||
tx: BoxedProgressSender<ConsistencyCheckProgress>,
|
|
||||||
) -> io::Result<()> {
|
|
||||||
self.inner.consistency_check(repair, tx).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn partial_blobs(&self) -> io::Result<DbIter<Hash>> {
|
|
||||||
self.inner.partial_blobs().await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn export(
|
|
||||||
&self,
|
|
||||||
hash: Hash,
|
|
||||||
target: PathBuf,
|
|
||||||
mode: ExportMode,
|
|
||||||
progress: ExportProgressCb,
|
|
||||||
) -> io::Result<()> {
|
|
||||||
self.inner.export(hash, target, mode, progress).await
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Store for ShapedStore {
|
|
||||||
async fn import_file(
|
|
||||||
&self,
|
|
||||||
data: PathBuf,
|
|
||||||
mode: ImportMode,
|
|
||||||
format: BlobFormat,
|
|
||||||
progress: impl ProgressSender<Msg = ImportProgress> + IdGenerator,
|
|
||||||
) -> io::Result<(TempTag, u64)> {
|
|
||||||
self.inner.import_file(data, mode, format, progress).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn import_bytes(&self, bytes: bytes::Bytes, format: BlobFormat) -> io::Result<TempTag> {
|
|
||||||
self.inner.import_bytes(bytes, format).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn import_stream(
|
|
||||||
&self,
|
|
||||||
data: impl futures_lite::Stream<Item = io::Result<bytes::Bytes>> + Send + Unpin + 'static,
|
|
||||||
format: BlobFormat,
|
|
||||||
progress: impl ProgressSender<Msg = ImportProgress> + IdGenerator,
|
|
||||||
) -> io::Result<(TempTag, u64)> {
|
|
||||||
self.inner.import_stream(data, format, progress).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn set_tag(&self, name: Tag, hash: HashAndFormat) -> io::Result<()> {
|
|
||||||
self.inner.set_tag(name, hash).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn rename_tag(&self, from: Tag, to: Tag) -> io::Result<()> {
|
|
||||||
self.inner.rename_tag(from, to).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn delete_tags(&self, from: Option<Tag>, to: Option<Tag>) -> io::Result<()> {
|
|
||||||
self.inner.delete_tags(from, to).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn create_tag(&self, hash: HashAndFormat) -> io::Result<Tag> {
|
|
||||||
self.inner.create_tag(hash).await
|
|
||||||
}
|
|
||||||
|
|
||||||
fn temp_tag(&self, value: HashAndFormat) -> TempTag {
|
|
||||||
self.inner.temp_tag(value)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn gc_run<G, Gut>(&self, config: iroh_blobs::store::GcConfig, protected_cb: G)
|
|
||||||
where
|
|
||||||
G: Fn() -> Gut,
|
|
||||||
Gut: Future<Output = std::collections::BTreeSet<Hash>> + Send,
|
|
||||||
{
|
|
||||||
self.inner.gc_run(config, protected_cb).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn delete(&self, hashes: Vec<Hash>) -> io::Result<()> {
|
|
||||||
self.inner.delete(hashes).await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn shutdown(&self) {
|
|
||||||
self.inner.shutdown().await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn sync(&self) -> io::Result<()> {
|
|
||||||
self.inner.sync().await
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
|
|||||||
+158
-190
@@ -1,24 +1,27 @@
|
|||||||
//! Blob store operations on top of the iroh-blobs persistent fs store.
|
//! Blob store operations on top of the iroh-blobs persistent fs store.
|
||||||
|
|
||||||
use std::collections::BTreeSet;
|
use std::collections::HashSet;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::sync::{Arc, OnceLock};
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
use anyhow::Context;
|
use anyhow::Context;
|
||||||
|
use iroh_blobs::api::blobs::{AddPathOptions, BlobStatus, ImportMode};
|
||||||
use iroh_blobs::format::collection::Collection;
|
use iroh_blobs::format::collection::Collection;
|
||||||
use iroh_blobs::hashseq::HashSeq;
|
use iroh_blobs::hashseq::HashSeq;
|
||||||
use iroh_blobs::store::{
|
use iroh_blobs::store::fs::options::{Options, PathOptions};
|
||||||
EntryStatus, ImportMode, Map, MapEntry, MapMut, ReadableStore, Store as _,
|
use iroh_blobs::store::fs::FsStore;
|
||||||
};
|
use iroh_blobs::store::{GcConfig, ProtectCb, ProtectOutcome};
|
||||||
use iroh_blobs::util::local_pool::{LocalPool, LocalPoolHandle};
|
|
||||||
use iroh_blobs::util::progress::IgnoreProgressSender;
|
|
||||||
use iroh_blobs::{BlobFormat, Hash, HashAndFormat};
|
use iroh_blobs::{BlobFormat, Hash, HashAndFormat};
|
||||||
use iroh_io::{AsyncSliceReader, AsyncSliceReaderExt};
|
use tracing::{debug, info, warn};
|
||||||
use tokio::io::AsyncWriteExt;
|
|
||||||
use tracing::{debug, info};
|
|
||||||
use varde_proto::MaterializeMode;
|
use varde_proto::MaterializeMode;
|
||||||
|
|
||||||
use crate::meta::StoredFormat;
|
use crate::meta::StoredFormat;
|
||||||
|
|
||||||
|
/// Snapshot of the gc roots (the pinned hashes with their formats),
|
||||||
|
/// consulted by the store's gc before every run.
|
||||||
|
pub type GcRootsFn = Arc<dyn Fn() -> Vec<HashAndFormat> + Send + Sync>;
|
||||||
|
|
||||||
/// Operation errors that map onto protocol error codes.
|
/// Operation errors that map onto protocol error codes.
|
||||||
#[derive(Debug, thiserror::Error)]
|
#[derive(Debug, thiserror::Error)]
|
||||||
pub enum OpError {
|
pub enum OpError {
|
||||||
@@ -34,64 +37,102 @@ pub enum OpError {
|
|||||||
Unimplemented(String),
|
Unimplemented(String),
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The daemon's blob store: an iroh-blobs fs store rooted at
|
/// Wrap an iroh-blobs API error as an internal error.
|
||||||
/// `<store_dir>/blobs`.
|
fn internal(e: impl std::error::Error + Send + Sync + 'static) -> OpError {
|
||||||
///
|
OpError::Internal(anyhow::Error::new(e))
|
||||||
/// Reads from store entries yield non-`Send` futures, so those operations
|
|
||||||
/// run on a dedicated [`LocalPool`] — the same pattern iroh-blobs itself
|
|
||||||
/// uses for its provider and GC tasks.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
pub struct BlobStore {
|
|
||||||
store: iroh_blobs::store::fs::Store,
|
|
||||||
data_dir: PathBuf,
|
|
||||||
pool: LocalPoolHandle,
|
|
||||||
// Owns the pool threads; dropped when the last clone goes away.
|
|
||||||
_pool: std::sync::Arc<LocalPool>,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Reading a blob entry for a streaming copy happens in chunks this size.
|
/// The daemon's blob store: an iroh-blobs fs store rooted at
|
||||||
const COPY_CHUNK: u64 = 1024 * 1024;
|
/// `<store_dir>/blobs`.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct BlobStore {
|
||||||
|
store: FsStore,
|
||||||
|
data_dir: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
impl BlobStore {
|
impl BlobStore {
|
||||||
pub async fn open(store_dir: &Path) -> anyhow::Result<BlobStore> {
|
/// Open the store. `gc_roots` supplies the pinned hashes the built-in
|
||||||
|
/// garbage collector must keep (temp tags and stored tags are
|
||||||
|
/// protected by the store itself); their hashseq children are
|
||||||
|
/// expanded here before every run.
|
||||||
|
pub async fn open(
|
||||||
|
store_dir: &Path,
|
||||||
|
gc_interval: Duration,
|
||||||
|
gc_roots: GcRootsFn,
|
||||||
|
) -> anyhow::Result<BlobStore> {
|
||||||
let blobs_dir = store_dir.join("blobs");
|
let blobs_dir = store_dir.join("blobs");
|
||||||
let store = iroh_blobs::store::fs::Store::load(&blobs_dir)
|
// The protect callback needs the store API to expand hashseq
|
||||||
|
// roots, but it is installed before the store exists — hence the
|
||||||
|
// cell, filled right after load.
|
||||||
|
let store_cell: Arc<OnceLock<iroh_blobs::api::Store>> = Arc::new(OnceLock::new());
|
||||||
|
let protect: ProtectCb = {
|
||||||
|
let cell = store_cell.clone();
|
||||||
|
Arc::new(move |live: &mut HashSet<Hash>| {
|
||||||
|
let cell = cell.clone();
|
||||||
|
let roots = gc_roots();
|
||||||
|
// The callback future must be Sync; the store API's
|
||||||
|
// futures are not, so the expansion runs on its own task.
|
||||||
|
Box::pin(async move {
|
||||||
|
let expanded = tokio::spawn(async move {
|
||||||
|
let mut hashes: Vec<Hash> = Vec::new();
|
||||||
|
for HashAndFormat { hash, format } in roots {
|
||||||
|
hashes.push(hash);
|
||||||
|
if format.is_raw() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let Some(store) = cell.get() else { continue };
|
||||||
|
match children_of(store, hash).await {
|
||||||
|
Ok(children) => hashes.extend(children),
|
||||||
|
Err(e) => {
|
||||||
|
warn!(hash = %hash.to_hex(), error = %e, "gc protect: expanding hashseq");
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Some(hashes)
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
match expanded {
|
||||||
|
Ok(Some(hashes)) => {
|
||||||
|
live.extend(hashes);
|
||||||
|
ProtectOutcome::Continue
|
||||||
|
}
|
||||||
|
// Confused about liveness: skip this run rather
|
||||||
|
// than sweep blobs that should be protected.
|
||||||
|
Ok(None) => ProtectOutcome::Abort,
|
||||||
|
Err(e) => {
|
||||||
|
warn!(error = %e, "gc protect task failed");
|
||||||
|
ProtectOutcome::Abort
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let options = Options {
|
||||||
|
path: PathOptions::new(&blobs_dir),
|
||||||
|
inline: Default::default(),
|
||||||
|
batch: Default::default(),
|
||||||
|
gc: Some(GcConfig {
|
||||||
|
interval: gc_interval,
|
||||||
|
add_protected: Some(protect),
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
let store = FsStore::load_with_opts(blobs_dir.join("blobs.db"), options)
|
||||||
.await
|
.await
|
||||||
.with_context(|| format!("opening blob store at {}", blobs_dir.display()))?;
|
.with_context(|| format!("opening blob store at {}", blobs_dir.display()))?;
|
||||||
let pool = LocalPool::default();
|
let _ = store_cell.set((*store).clone());
|
||||||
Ok(BlobStore {
|
Ok(BlobStore {
|
||||||
store,
|
store,
|
||||||
data_dir: blobs_dir.join("data"),
|
data_dir: blobs_dir.join("data"),
|
||||||
pool: pool.handle().clone(),
|
|
||||||
_pool: std::sync::Arc::new(pool),
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Run a non-`Send` store operation on the local pool.
|
/// Access to the underlying iroh-blobs store API (used by the
|
||||||
async fn on_pool<T, F, Fut>(&self, f: F) -> Result<T, OpError>
|
/// transfer layer). `FsStore` derefs to the API handle.
|
||||||
where
|
pub fn api(&self) -> &iroh_blobs::api::Store {
|
||||||
F: FnOnce(BlobStore) -> Fut + Send + 'static,
|
|
||||||
Fut: std::future::Future<Output = Result<T, OpError>> + 'static,
|
|
||||||
T: Send + 'static,
|
|
||||||
{
|
|
||||||
let this = self.clone();
|
|
||||||
self.pool
|
|
||||||
.spawn(move || f(this))
|
|
||||||
.await
|
|
||||||
.map_err(|e| OpError::Internal(anyhow::anyhow!("local pool: {e}")))?
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Access to the underlying iroh-blobs store (used by the transfer
|
|
||||||
/// layer).
|
|
||||||
pub fn inner(&self) -> &iroh_blobs::store::fs::Store {
|
|
||||||
&self.store
|
&self.store
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The local pool the transfer layer shares for non-`Send` blob work.
|
|
||||||
pub fn pool_handle(&self) -> LocalPoolHandle {
|
|
||||||
self.pool.clone()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Import a file or directory. Returns the root hash, total imported
|
/// Import a file or directory. Returns the root hash, total imported
|
||||||
/// bytes, and the resulting format (`Raw` for files, `HashSeq` for
|
/// bytes, and the resulting format (`Raw` for files, `HashSeq` for
|
||||||
/// directories).
|
/// directories).
|
||||||
@@ -110,7 +151,7 @@ impl BlobStore {
|
|||||||
if meta.is_file() {
|
if meta.is_file() {
|
||||||
let (tag, size) = self.import_one(path.to_owned()).await?;
|
let (tag, size) = self.import_one(path.to_owned()).await?;
|
||||||
info!(hash = %tag.hash().to_hex(), size, "imported file");
|
info!(hash = %tag.hash().to_hex(), size, "imported file");
|
||||||
return Ok((*tag.hash(), size, StoredFormat::Raw));
|
return Ok((tag.hash(), size, StoredFormat::Raw));
|
||||||
}
|
}
|
||||||
if !meta.is_dir() {
|
if !meta.is_dir() {
|
||||||
return Err(OpError::InvalidArgument(format!(
|
return Err(OpError::InvalidArgument(format!(
|
||||||
@@ -145,27 +186,40 @@ impl BlobStore {
|
|||||||
}
|
}
|
||||||
let collection: Collection = children
|
let collection: Collection = children
|
||||||
.iter()
|
.iter()
|
||||||
.map(|(name, tag)| (name.clone(), *tag.hash()))
|
.map(|(name, tag)| (name.clone(), tag.hash()))
|
||||||
.collect();
|
.collect();
|
||||||
let root_tag = collection
|
let root_tag = collection
|
||||||
.store(&self.store)
|
.store(&self.store)
|
||||||
.await
|
.await
|
||||||
.context("storing collection")?;
|
.context("storing collection")?;
|
||||||
let root = *root_tag.hash();
|
let root = root_tag.hash();
|
||||||
info!(hash = %root.to_hex(), files = children.len(), total, "imported directory");
|
info!(hash = %root.to_hex(), files = children.len(), total, "imported directory");
|
||||||
Ok((root, total, StoredFormat::HashSeq))
|
Ok((root, total, StoredFormat::HashSeq))
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn import_one(&self, path: PathBuf) -> Result<(iroh_blobs::TempTag, u64), OpError> {
|
async fn import_one(
|
||||||
let (tag, size) = self
|
&self,
|
||||||
|
path: PathBuf,
|
||||||
|
) -> Result<(iroh_blobs::api::TempTag, u64), OpError> {
|
||||||
|
let tag = self
|
||||||
.store
|
.store
|
||||||
.import_file(
|
.add_path_with_opts(AddPathOptions {
|
||||||
path,
|
path,
|
||||||
ImportMode::Copy,
|
format: BlobFormat::Raw,
|
||||||
BlobFormat::Raw,
|
mode: ImportMode::Copy,
|
||||||
IgnoreProgressSender::default(),
|
})
|
||||||
)
|
.temp_tag()
|
||||||
.await?;
|
.await
|
||||||
|
.map_err(internal)?;
|
||||||
|
let size = match self.store.blobs().status(tag.hash()).await.map_err(internal)? {
|
||||||
|
BlobStatus::Complete { size } => size,
|
||||||
|
_ => {
|
||||||
|
return Err(OpError::Internal(anyhow::anyhow!(
|
||||||
|
"freshly imported blob {} is not complete",
|
||||||
|
tag.hash().to_hex()
|
||||||
|
)))
|
||||||
|
}
|
||||||
|
};
|
||||||
Ok((tag, size))
|
Ok((tag, size))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -184,25 +238,11 @@ impl BlobStore {
|
|||||||
dest.display()
|
dest.display()
|
||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
let dest = dest.to_owned();
|
|
||||||
self.on_pool(
|
|
||||||
move |this| async move { this.materialize_local(hash, format, &dest, mode).await },
|
|
||||||
)
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn materialize_local(
|
|
||||||
&self,
|
|
||||||
hash: Hash,
|
|
||||||
format: StoredFormat,
|
|
||||||
dest: &Path,
|
|
||||||
mode: MaterializeMode,
|
|
||||||
) -> Result<(u64, bool), OpError> {
|
|
||||||
let allow_reflink = matches!(mode, MaterializeMode::ReflinkOrCopy);
|
let allow_reflink = matches!(mode, MaterializeMode::ReflinkOrCopy);
|
||||||
match format {
|
match format {
|
||||||
StoredFormat::Raw => self.export_blob(hash, dest, allow_reflink).await,
|
StoredFormat::Raw => self.export_blob(hash, dest, allow_reflink).await,
|
||||||
StoredFormat::HashSeq => {
|
StoredFormat::HashSeq => {
|
||||||
let collection = Collection::load_db(&self.store, &hash)
|
let collection = Collection::load(hash, self.api())
|
||||||
.await
|
.await
|
||||||
.map_err(|e| OpError::NotFound(format!("loading collection: {e}")))?;
|
.map_err(|e| OpError::NotFound(format!("loading collection: {e}")))?;
|
||||||
let mut total = 0u64;
|
let mut total = 0u64;
|
||||||
@@ -222,25 +262,30 @@ impl BlobStore {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Export one blob. Tries `FICLONE` via reflink first (zero-copy on
|
/// Export one blob. Tries `FICLONE` via reflink first (zero-copy on
|
||||||
/// btrfs/XFS when store and dest share a filesystem), falls back to a
|
/// btrfs/XFS when store and dest share a filesystem), falls back to
|
||||||
/// streaming copy. Never hardlinks: store files must stay immutable.
|
/// the store's export (a copy). Never hardlinks: store files must
|
||||||
|
/// stay immutable.
|
||||||
async fn export_blob(
|
async fn export_blob(
|
||||||
&self,
|
&self,
|
||||||
hash: Hash,
|
hash: Hash,
|
||||||
dest: &Path,
|
dest: &Path,
|
||||||
allow_reflink: bool,
|
allow_reflink: bool,
|
||||||
) -> Result<(u64, bool), OpError> {
|
) -> Result<(u64, bool), OpError> {
|
||||||
let entry =
|
let size = match self.store.blobs().status(hash).await.map_err(internal)? {
|
||||||
self.store.get(&hash).await?.ok_or_else(|| {
|
BlobStatus::Complete { size } => size,
|
||||||
OpError::NotFound(format!("{} is not in the store", hash.to_hex()))
|
BlobStatus::Partial { .. } => {
|
||||||
})?;
|
|
||||||
if !entry.is_complete() {
|
|
||||||
return Err(OpError::NotFound(format!(
|
return Err(OpError::NotFound(format!(
|
||||||
"{} is only partially present",
|
"{} is only partially present",
|
||||||
hash.to_hex()
|
hash.to_hex()
|
||||||
)));
|
)))
|
||||||
}
|
}
|
||||||
let size = entry.size().value();
|
BlobStatus::NotFound => {
|
||||||
|
return Err(OpError::NotFound(format!(
|
||||||
|
"{} is not in the store",
|
||||||
|
hash.to_hex()
|
||||||
|
)))
|
||||||
|
}
|
||||||
|
};
|
||||||
if let Some(parent) = dest.parent() {
|
if let Some(parent) = dest.parent() {
|
||||||
tokio::fs::create_dir_all(parent).await?;
|
tokio::fs::create_dir_all(parent).await?;
|
||||||
}
|
}
|
||||||
@@ -272,129 +317,52 @@ impl BlobStore {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut reader = entry.data_reader();
|
let bytes = self
|
||||||
let mut file = tokio::fs::File::create(dest).await?;
|
.store
|
||||||
let mut offset = 0u64;
|
.blobs()
|
||||||
while offset < size {
|
.export(hash, dest)
|
||||||
let len = (size - offset).min(COPY_CHUNK) as usize;
|
.await
|
||||||
let chunk = reader.read_at(offset, len).await?;
|
.map_err(internal)?;
|
||||||
if chunk.is_empty() {
|
Ok((bytes, false))
|
||||||
return Err(OpError::Io(std::io::Error::new(
|
|
||||||
std::io::ErrorKind::UnexpectedEof,
|
|
||||||
format!("blob {} truncated at {offset}", hash.to_hex()),
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
file.write_all(&chunk).await?;
|
|
||||||
offset += chunk.len() as u64;
|
|
||||||
}
|
|
||||||
file.flush().await?;
|
|
||||||
Ok((size, false))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Presence of `hash`: (have_bytes, total_bytes, complete).
|
/// Presence of `hash`: (have_bytes, total_bytes, complete).
|
||||||
pub async fn presence(&self, hash: Hash) -> Result<(u64, Option<u64>, bool), OpError> {
|
pub async fn presence(&self, hash: Hash) -> Result<(u64, Option<u64>, bool), OpError> {
|
||||||
match self.store.entry_status(&hash).await? {
|
match self.store.blobs().status(hash).await.map_err(internal)? {
|
||||||
EntryStatus::NotFound => Ok((0, None, false)),
|
BlobStatus::NotFound => Ok((0, None, false)),
|
||||||
EntryStatus::Partial => {
|
|
||||||
let total = self.store.get(&hash).await?.map(|e| e.size().value());
|
|
||||||
// Valid-range accounting for partials arrives with the
|
// Valid-range accounting for partials arrives with the
|
||||||
// transfer milestone; absence of data is the safe report.
|
// transfer milestone; absence of data is the safe report.
|
||||||
Ok((0, total, false))
|
BlobStatus::Partial { size } => Ok((0, size, false)),
|
||||||
}
|
BlobStatus::Complete { size } => Ok((size, Some(size), true)),
|
||||||
EntryStatus::Complete => {
|
|
||||||
let entry = self
|
|
||||||
.store
|
|
||||||
.get(&hash)
|
|
||||||
.await?
|
|
||||||
.ok_or_else(|| OpError::NotFound(hash.to_hex().to_string()))?;
|
|
||||||
let size = entry.size().value();
|
|
||||||
Ok((size, Some(size), true))
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The child hashes of a complete HashSeq root (empty for absent or
|
/// The child hashes of a complete HashSeq root (empty for absent or
|
||||||
/// partial roots).
|
/// partial roots).
|
||||||
pub async fn hashseq_children(&self, root: Hash) -> Result<Vec<Hash>, OpError> {
|
pub async fn hashseq_children(&self, root: Hash) -> Result<Vec<Hash>, OpError> {
|
||||||
self.on_pool(move |this| async move {
|
children_of(self.api(), root).await
|
||||||
let Some(entry) = this.store.get(&root).await? else {
|
|
||||||
return Ok(Vec::new());
|
|
||||||
};
|
|
||||||
if !entry.is_complete() {
|
|
||||||
return Ok(Vec::new());
|
|
||||||
}
|
|
||||||
let mut reader = entry.data_reader();
|
|
||||||
let bytes = reader.read_to_end().await?;
|
|
||||||
let seq = HashSeq::try_from(bytes)
|
|
||||||
.map_err(|e| OpError::Internal(anyhow::anyhow!("invalid hashseq: {e}")))?;
|
|
||||||
Ok(seq.iter().collect())
|
|
||||||
})
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Drop every blob not reachable from `roots` (mark and sweep).
|
|
||||||
/// In-flight imports are protected by their temp tags; tags stored in
|
|
||||||
/// the blob database are honored too.
|
|
||||||
pub async fn gc(&self, roots: Vec<HashAndFormat>) -> Result<u64, OpError> {
|
|
||||||
self.on_pool(move |this| async move { this.gc_local(roots).await })
|
|
||||||
.await
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn gc_local(&self, roots: Vec<HashAndFormat>) -> Result<u64, OpError> {
|
|
||||||
let mut live: BTreeSet<Hash> = BTreeSet::new();
|
|
||||||
let mut all_roots = roots;
|
|
||||||
all_roots.extend(self.store.temp_tags());
|
|
||||||
for item in self.store.tags(None, None).await.context(TAGS_CONTEXT)? {
|
|
||||||
let (_name, haf) = item.context(TAGS_CONTEXT)?;
|
|
||||||
all_roots.push(haf);
|
|
||||||
}
|
|
||||||
|
|
||||||
for HashAndFormat { hash, format } in all_roots {
|
|
||||||
if !live.insert(hash) || format.is_raw() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
// HashSeq root: its children are live too. A partial root
|
|
||||||
// can't be expanded; its bytes are still protected.
|
|
||||||
let Some(entry) = self.store.get(&hash).await? else {
|
|
||||||
continue;
|
|
||||||
};
|
|
||||||
if !entry.is_complete() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
let mut reader = entry.data_reader();
|
|
||||||
let bytes = reader.read_to_end().await?;
|
|
||||||
let seq = HashSeq::try_from(bytes)
|
|
||||||
.map_err(|e| OpError::Internal(anyhow::anyhow!("invalid hashseq: {e}")))?;
|
|
||||||
live.extend(seq.iter());
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut doomed = Vec::new();
|
|
||||||
for hash in self
|
|
||||||
.store
|
|
||||||
.blobs()
|
|
||||||
.await?
|
|
||||||
.chain(self.store.partial_blobs().await?)
|
|
||||||
{
|
|
||||||
let hash = hash?;
|
|
||||||
if !live.contains(&hash) {
|
|
||||||
doomed.push(hash);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let removed = doomed.len() as u64;
|
|
||||||
if !doomed.is_empty() {
|
|
||||||
self.store.delete(doomed).await?;
|
|
||||||
}
|
|
||||||
info!(removed, live = live.len(), "gc done");
|
|
||||||
Ok(removed)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Flush and shut down the store actor.
|
/// Flush and shut down the store actor.
|
||||||
pub async fn shutdown(&self) {
|
pub async fn shutdown(&self) {
|
||||||
self.store.shutdown().await;
|
if let Err(e) = self.store.shutdown().await {
|
||||||
|
debug!(error = %e, "store shutdown");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const TAGS_CONTEXT: &str = "listing tags";
|
/// The child hashes of a complete HashSeq root (empty for absent or
|
||||||
|
/// partial roots).
|
||||||
|
async fn children_of(store: &iroh_blobs::api::Store, root: Hash) -> Result<Vec<Hash>, OpError> {
|
||||||
|
match store.blobs().status(root).await.map_err(internal)? {
|
||||||
|
BlobStatus::Complete { .. } => {}
|
||||||
|
_ => return Ok(Vec::new()),
|
||||||
|
}
|
||||||
|
let bytes = store.blobs().get_bytes(root).await.map_err(internal)?;
|
||||||
|
let seq = HashSeq::try_from(bytes)
|
||||||
|
.map_err(|e| OpError::Internal(anyhow::anyhow!("invalid hashseq: {e}")))?;
|
||||||
|
Ok(seq.iter().collect())
|
||||||
|
}
|
||||||
|
|
||||||
/// Recursively collect regular files under `dir` as (relative-name, path),
|
/// Recursively collect regular files under `dir` as (relative-name, path),
|
||||||
/// sorted for deterministic collection hashes. Symlinks are followed for
|
/// sorted for deterministic collection hashes. Symlinks are followed for
|
||||||
|
|||||||
+123
-178
@@ -1,25 +1,20 @@
|
|||||||
//! iroh endpoint, blob provider and downloader.
|
//! iroh endpoint, blob provider and downloader.
|
||||||
//!
|
//!
|
||||||
//! This module is the transport seam: rate limiting lives in
|
//! This module is the transport seam: trust gating and rate limiting
|
||||||
//! [`crate::shaped::ShapedStore`] wired up here, and a scavenger
|
//! live in [`crate::shaped`]'s provider event handler wired up here, and
|
||||||
//! congestion controller would later slot in at the same place without
|
//! a scavenger congestion controller would later slot in at the same
|
||||||
//! touching the daemon logic above.
|
//! place without touching the daemon logic above.
|
||||||
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
use anyhow::{Context, Result};
|
use anyhow::{Context, Result};
|
||||||
use iroh::discovery::mdns::MdnsDiscovery;
|
use iroh::address_lookup::memory::MemoryLookup;
|
||||||
use iroh::endpoint::Connection;
|
use iroh::protocol::Router;
|
||||||
use iroh::protocol::{ProtocolHandler, Router};
|
use iroh::{Endpoint, EndpointAddr, EndpointId, RelayMode, SecretKey};
|
||||||
use iroh::{Endpoint, NodeAddr, NodeId, RelayMode, SecretKey};
|
use iroh_blobs::api::downloader::{DownloadOptions, Downloader, Shuffled, SplitStrategy};
|
||||||
use iroh_blobs::downloader::{DownloadRequest, Downloader};
|
use iroh_blobs::{BlobFormat, BlobsProtocol, Hash, HashAndFormat};
|
||||||
use iroh_blobs::get::db::DownloadProgress;
|
use iroh_mdns_address_lookup::{DiscoveryEvent, MdnsAddressLookup};
|
||||||
use iroh_blobs::provider::{handle_connection, CustomEventSender, EventSender};
|
|
||||||
use iroh_blobs::util::local_pool::LocalPoolHandle;
|
|
||||||
use iroh_blobs::util::progress::AsyncChannelProgressSender;
|
|
||||||
use iroh_blobs::{BlobFormat, Hash, HashAndFormat};
|
|
||||||
use n0_future::boxed::BoxFuture;
|
|
||||||
use n0_future::StreamExt;
|
use n0_future::StreamExt;
|
||||||
use tokio::sync::{broadcast, mpsc};
|
use tokio::sync::{broadcast, mpsc};
|
||||||
use tracing::{debug, info, warn};
|
use tracing::{debug, info, warn};
|
||||||
@@ -28,7 +23,7 @@ use varde_proto::{Direction, Event};
|
|||||||
use crate::config::Config;
|
use crate::config::Config;
|
||||||
use crate::meta::Meta;
|
use crate::meta::Meta;
|
||||||
use crate::metered::MeteredState;
|
use crate::metered::MeteredState;
|
||||||
use crate::shaped::{ServeFilter, ShapedStore, TokenBucket};
|
use crate::shaped::{Gate, ServeFilter, TokenBucket};
|
||||||
use crate::store::BlobStore;
|
use crate::store::BlobStore;
|
||||||
|
|
||||||
/// The network side of the daemon: one iroh endpoint serving the blob
|
/// The network side of the daemon: one iroh endpoint serving the blob
|
||||||
@@ -39,71 +34,13 @@ pub struct Transfer {
|
|||||||
endpoint: Endpoint,
|
endpoint: Endpoint,
|
||||||
router: Router,
|
router: Router,
|
||||||
downloader: Downloader,
|
downloader: Downloader,
|
||||||
|
/// Address book for provider addresses learned out-of-band (ticket
|
||||||
|
/// imports): the downloader dials by endpoint id and resolves
|
||||||
|
/// addresses through the endpoint's address lookup services.
|
||||||
|
known_addrs: MemoryLookup,
|
||||||
events: broadcast::Sender<Event>,
|
events: broadcast::Sender<Event>,
|
||||||
metered: MeteredState,
|
metered: MeteredState,
|
||||||
discovery_rx: std::sync::Mutex<Option<mpsc::Receiver<NodeId>>>,
|
discovery_rx: std::sync::Mutex<Option<mpsc::Receiver<EndpointId>>>,
|
||||||
}
|
|
||||||
|
|
||||||
/// Serves blob requests with trust gating: trusted peers see the whole
|
|
||||||
/// store, untrusted peers see only the openly-served subset (and get
|
|
||||||
/// "not found" for the rest). With serving disabled (upload cap 0) every
|
|
||||||
/// incoming connection is closed immediately.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
struct GatedProvider {
|
|
||||||
full_view: ShapedStore,
|
|
||||||
open_view: ShapedStore,
|
|
||||||
meta: Arc<Meta>,
|
|
||||||
events: EventSender,
|
|
||||||
pool: LocalPoolHandle,
|
|
||||||
serving_enabled: bool,
|
|
||||||
metered: MeteredState,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl ProtocolHandler for GatedProvider {
|
|
||||||
fn accept(&self, connection: Connection) -> BoxFuture<Result<()>> {
|
|
||||||
let this = self.clone();
|
|
||||||
Box::pin(async move {
|
|
||||||
let remote = connection.remote_node_id()?;
|
|
||||||
if !this.serving_enabled || this.metered.is_metered() {
|
|
||||||
debug!(%remote, "serving disabled or metered, closing connection");
|
|
||||||
connection.close(0u32.into(), b"serving disabled");
|
|
||||||
return Ok(());
|
|
||||||
}
|
|
||||||
let trusted = this.meta.is_trusted(&remote.to_string());
|
|
||||||
debug!(%remote, trusted, "incoming blobs connection");
|
|
||||||
if trusted {
|
|
||||||
handle_connection(connection, this.full_view, this.events, this.pool).await;
|
|
||||||
} else {
|
|
||||||
handle_connection(connection, this.open_view, this.events, this.pool).await;
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Forwards provider transfer events into the daemon event stream.
|
|
||||||
#[derive(Debug)]
|
|
||||||
struct UploadEvents(broadcast::Sender<Event>);
|
|
||||||
|
|
||||||
impl CustomEventSender for UploadEvents {
|
|
||||||
fn send(&self, event: iroh_blobs::provider::Event) -> BoxFuture<()> {
|
|
||||||
self.try_send(event);
|
|
||||||
Box::pin(async {})
|
|
||||||
}
|
|
||||||
|
|
||||||
fn try_send(&self, event: iroh_blobs::provider::Event) {
|
|
||||||
if let iroh_blobs::provider::Event::TransferProgress {
|
|
||||||
hash, end_offset, ..
|
|
||||||
} = event
|
|
||||||
{
|
|
||||||
let _ = self.0.send(Event::TransferProgress {
|
|
||||||
hash: hash.to_hex().to_string(),
|
|
||||||
direction: Direction::Upload,
|
|
||||||
bytes_done: end_offset,
|
|
||||||
bytes_total: None,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Transfer {
|
impl Transfer {
|
||||||
@@ -117,60 +54,71 @@ impl Transfer {
|
|||||||
store: &BlobStore,
|
store: &BlobStore,
|
||||||
meta: Arc<Meta>,
|
meta: Arc<Meta>,
|
||||||
open_filter: ServeFilter,
|
open_filter: ServeFilter,
|
||||||
pool: LocalPoolHandle,
|
|
||||||
events: broadcast::Sender<Event>,
|
events: broadcast::Sender<Event>,
|
||||||
metered: MeteredState,
|
metered: MeteredState,
|
||||||
) -> Result<Transfer> {
|
) -> Result<Transfer> {
|
||||||
let secret = load_or_create_secret(&config.store_dir.join("secret.key"))?;
|
let secret = load_or_create_secret(&config.store_dir.join("secret.key"))?;
|
||||||
let mut builder = Endpoint::builder().secret_key(secret.clone());
|
// With wan_upload the n0 defaults apply (their relays and DNS
|
||||||
if !config.wan_upload {
|
// lookup, matching the pre-1.0 default relay mode); otherwise the
|
||||||
builder = builder.relay_mode(RelayMode::Disabled);
|
// endpoint gets no external services at all.
|
||||||
}
|
let builder = if config.wan_upload {
|
||||||
|
Endpoint::builder(iroh::endpoint::presets::N0)
|
||||||
|
} else {
|
||||||
|
Endpoint::builder(iroh::endpoint::presets::Minimal).relay_mode(RelayMode::Disabled)
|
||||||
|
};
|
||||||
|
let endpoint = builder
|
||||||
|
.secret_key(secret.clone())
|
||||||
|
.bind()
|
||||||
|
.await
|
||||||
|
.context("binding iroh endpoint")?;
|
||||||
|
info!(endpoint_id = %endpoint.id(), discovery = config.discovery, "endpoint up");
|
||||||
|
|
||||||
|
let known_addrs = MemoryLookup::with_provenance("varde-ticket");
|
||||||
|
endpoint
|
||||||
|
.address_lookup()
|
||||||
|
.context("endpoint address lookup services")?
|
||||||
|
.add(known_addrs.clone());
|
||||||
|
|
||||||
// LAN discovery: advertise under iroh's mDNS-style local
|
// LAN discovery: advertise under iroh's mDNS-style local
|
||||||
// discovery service (identifiable `_iroh` mdns records, not
|
// discovery service (identifiable `_iroh` mdns records, not
|
||||||
// anonymous noise). Discovered node ids stream to the daemon,
|
// anonymous noise). Discovered endpoint ids stream to the daemon,
|
||||||
// which decides what to do based on trust.
|
// which decides what to do based on trust.
|
||||||
let (discovery_tx, discovery_rx) = mpsc::channel(64);
|
let (discovery_tx, discovery_rx) = mpsc::channel(64);
|
||||||
if config.discovery {
|
if config.discovery {
|
||||||
let mdns =
|
let mdns = MdnsAddressLookup::builder()
|
||||||
MdnsDiscovery::new(secret.public()).context("starting mdns local discovery")?;
|
.build(endpoint.id())
|
||||||
if let Some(mut stream) = iroh::discovery::Discovery::subscribe(&mdns) {
|
.context("starting mdns local discovery")?;
|
||||||
|
let mut sightings = mdns.subscribe().await;
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
while let Some(item) = stream.next().await {
|
while let Some(event) = sightings.next().await {
|
||||||
if discovery_tx.send(item.node_id()).await.is_err() {
|
let DiscoveryEvent::Discovered { endpoint_info, .. } = event else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if discovery_tx.send(endpoint_info.endpoint_id).await.is_err() {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
endpoint
|
||||||
|
.address_lookup()
|
||||||
|
.context("endpoint address lookup services")?
|
||||||
|
.add(mdns);
|
||||||
}
|
}
|
||||||
builder = builder.discovery(Box::new(mdns));
|
|
||||||
}
|
|
||||||
|
|
||||||
let endpoint = builder.bind().await.context("binding iroh endpoint")?;
|
|
||||||
info!(node_id = %endpoint.node_id(), discovery = config.discovery, "endpoint up");
|
|
||||||
|
|
||||||
// Best-effort low-priority marking of our UDP traffic.
|
// Best-effort low-priority marking of our UDP traffic.
|
||||||
let (v4, v6) = endpoint.bound_sockets();
|
crate::dscp::mark_endpoint_sockets(&endpoint.bound_sockets());
|
||||||
let bound: Vec<std::net::SocketAddr> = std::iter::once(v4).chain(v6).collect();
|
|
||||||
crate::dscp::mark_endpoint_sockets(&bound);
|
|
||||||
|
|
||||||
let upload_bucket = TokenBucket::new(config.max_upload_bytes_per_sec);
|
let provider_events = crate::shaped::provider_events(Gate {
|
||||||
let download_bucket = TokenBucket::new(config.max_download_bytes_per_sec);
|
|
||||||
let shaped = ShapedStore::new(store.inner().clone(), upload_bucket, download_bucket);
|
|
||||||
|
|
||||||
let provider_events = EventSender::new(Some(Arc::new(UploadEvents(events.clone()))));
|
|
||||||
let provider = GatedProvider {
|
|
||||||
full_view: shaped.clone(),
|
|
||||||
open_view: shaped.filtered(open_filter),
|
|
||||||
meta,
|
meta,
|
||||||
events: provider_events,
|
open_filter,
|
||||||
pool: pool.clone(),
|
upload: TokenBucket::new(config.max_upload_bytes_per_sec),
|
||||||
serving_enabled: config.max_upload_bytes_per_sec > 0,
|
serving_enabled: config.max_upload_bytes_per_sec > 0,
|
||||||
metered: metered.clone(),
|
metered: metered.clone(),
|
||||||
};
|
events: events.clone(),
|
||||||
|
});
|
||||||
|
let provider = BlobsProtocol::new(store.api(), Some(provider_events));
|
||||||
|
|
||||||
let downloader = Downloader::new(shaped, endpoint.clone(), pool);
|
let downloader = store.api().downloader(&endpoint);
|
||||||
let router = Router::builder(endpoint.clone())
|
let router = Router::builder(endpoint.clone())
|
||||||
.accept(iroh_blobs::ALPN, provider)
|
.accept(iroh_blobs::ALPN, provider)
|
||||||
.spawn();
|
.spawn();
|
||||||
@@ -179,15 +127,16 @@ impl Transfer {
|
|||||||
endpoint,
|
endpoint,
|
||||||
router,
|
router,
|
||||||
downloader,
|
downloader,
|
||||||
|
known_addrs,
|
||||||
events,
|
events,
|
||||||
metered,
|
metered,
|
||||||
discovery_rx: std::sync::Mutex::new(Some(discovery_rx)),
|
discovery_rx: std::sync::Mutex::new(Some(discovery_rx)),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Our stable node id.
|
/// Our stable endpoint id.
|
||||||
pub fn node_id(&self) -> String {
|
pub fn node_id(&self) -> String {
|
||||||
self.endpoint.node_id().to_string()
|
self.endpoint.id().to_string()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The endpoint's identity key (also signs our announcements).
|
/// The endpoint's identity key (also signs our announcements).
|
||||||
@@ -195,69 +144,90 @@ impl Transfer {
|
|||||||
self.endpoint.secret_key().clone()
|
self.endpoint.secret_key().clone()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Take the stream of locally discovered node ids. Yields each
|
/// Take the stream of locally discovered endpoint ids. Yields each
|
||||||
/// sighting (mdns re-announces periodically); the daemon dedupes.
|
/// sighting (mdns re-announces periodically); the daemon dedupes.
|
||||||
/// Callable once; returns None afterwards or with discovery off.
|
/// Callable once; returns None afterwards or with discovery off.
|
||||||
pub fn take_discovery(&self) -> Option<mpsc::Receiver<NodeId>> {
|
pub fn take_discovery(&self) -> Option<mpsc::Receiver<EndpointId>> {
|
||||||
self.discovery_rx.lock().expect("discovery lock").take()
|
self.discovery_rx.lock().expect("discovery lock").take()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Produce a ticket for out-of-band sharing of `content`.
|
/// Produce a ticket for out-of-band sharing of `content`.
|
||||||
pub async fn export_ticket(&self, content: HashAndFormat) -> Result<String> {
|
pub async fn export_ticket(&self, content: HashAndFormat) -> Result<String> {
|
||||||
let addr = self
|
let addr = self.endpoint.addr();
|
||||||
.endpoint
|
anyhow::ensure!(
|
||||||
.node_addr()
|
!addr.addrs.is_empty(),
|
||||||
.await
|
"endpoint has no dialable addresses yet"
|
||||||
.context("waiting for endpoint address")?;
|
);
|
||||||
let ticket = iroh_blobs::ticket::BlobTicket::new(addr, content.hash, content.format)?;
|
let ticket = iroh_blobs::ticket::BlobTicket::new(addr, content.hash, content.format);
|
||||||
Ok(ticket.to_string())
|
Ok(ticket.to_string())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Queue a background fetch of `content` from `providers`. Emits
|
/// Queue a background fetch of `content` from `providers`. Emits
|
||||||
/// progress events and [`Event::PinComplete`] when fully verified.
|
/// progress events and [`Event::PinComplete`] when fully verified.
|
||||||
pub fn spawn_fetch(&self, content: HashAndFormat, providers: Vec<NodeAddr>) {
|
pub fn spawn_fetch(&self, content: HashAndFormat, providers: Vec<EndpointAddr>) {
|
||||||
if self.metered.is_metered() {
|
if self.metered.is_metered() {
|
||||||
// The pin stays recorded; auto-sync retries once unmetered.
|
// The pin stays recorded; auto-sync retries once unmetered.
|
||||||
info!(hash = %content.hash.to_hex(), "metered connection: fetch deferred");
|
info!(hash = %content.hash.to_hex(), "metered connection: fetch deferred");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
// The downloader dials by EndpointId; teach the endpoint any
|
||||||
|
// direct addresses we know (ticket-embedded ones). Id-only
|
||||||
|
// provider entries resolve via discovery instead.
|
||||||
|
let mut ids = Vec::with_capacity(providers.len());
|
||||||
|
for provider in providers {
|
||||||
|
ids.push(provider.id);
|
||||||
|
if !provider.addrs.is_empty() {
|
||||||
|
self.known_addrs.add_endpoint_info(provider);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
let downloader = self.downloader.clone();
|
let downloader = self.downloader.clone();
|
||||||
let events = self.events.clone();
|
let events = self.events.clone();
|
||||||
let endpoint = self.endpoint.clone();
|
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
// The downloader dials by NodeId; the endpoint must be taught
|
let progress = downloader.download_with_opts(DownloadOptions::new(
|
||||||
// any direct addresses we know (ticket-embedded ones), or the
|
content,
|
||||||
// dial fails with "no addressing information". Id-only
|
Shuffled::new(ids),
|
||||||
// provider entries resolve via discovery instead.
|
SplitStrategy::None,
|
||||||
for provider in &providers {
|
|
||||||
if !provider.direct_addresses.is_empty() || provider.relay_url.is_some() {
|
|
||||||
if let Err(e) = endpoint.add_node_addr(provider.clone()) {
|
|
||||||
debug!(node = %provider.node_id, error = %e, "adding provider address");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let (progress_tx, progress_rx) = async_channel::bounded(64);
|
|
||||||
tokio::spawn(forward_download_progress(
|
|
||||||
content.hash,
|
|
||||||
progress_rx,
|
|
||||||
events.clone(),
|
|
||||||
));
|
));
|
||||||
|
let mut stream = match progress.stream().await {
|
||||||
let request = DownloadRequest::new(content, providers)
|
Ok(stream) => stream,
|
||||||
.progress_sender(AsyncChannelProgressSender::new(progress_tx));
|
Err(e) => {
|
||||||
let handle = downloader.queue(request).await;
|
warn!(hash = %content.hash.to_hex(), error = %e, "fetch failed to start");
|
||||||
match handle.await {
|
return;
|
||||||
Ok(_stats) => {
|
}
|
||||||
|
};
|
||||||
|
use iroh_blobs::api::downloader::DownloadProgressItem;
|
||||||
|
let mut failed = false;
|
||||||
|
while let Some(item) = stream.next().await {
|
||||||
|
match item {
|
||||||
|
DownloadProgressItem::Progress(bytes_done) => {
|
||||||
|
let _ = events.send(Event::TransferProgress {
|
||||||
|
hash: content.hash.to_hex().to_string(),
|
||||||
|
direction: Direction::Download,
|
||||||
|
bytes_done,
|
||||||
|
bytes_total: None,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
DownloadProgressItem::ProviderFailed { id, .. } => {
|
||||||
|
debug!(hash = %content.hash.to_hex(), provider = %id, "provider failed");
|
||||||
|
}
|
||||||
|
DownloadProgressItem::Error(e) => {
|
||||||
|
warn!(hash = %content.hash.to_hex(), error = %e, "fetch failed");
|
||||||
|
failed = true;
|
||||||
|
}
|
||||||
|
DownloadProgressItem::DownloadError => {
|
||||||
|
warn!(hash = %content.hash.to_hex(), "fetch failed");
|
||||||
|
failed = true;
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !failed {
|
||||||
info!(hash = %content.hash.to_hex(), "fetch complete");
|
info!(hash = %content.hash.to_hex(), "fetch complete");
|
||||||
let _ = events.send(Event::PinComplete {
|
let _ = events.send(Event::PinComplete {
|
||||||
hash: content.hash.to_hex().to_string(),
|
hash: content.hash.to_hex().to_string(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
Err(e) => {
|
|
||||||
warn!(hash = %content.hash.to_hex(), error = %e, "fetch failed");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -270,37 +240,12 @@ impl Transfer {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Translate iroh-blobs download progress into daemon transfer events.
|
|
||||||
async fn forward_download_progress(
|
|
||||||
root: Hash,
|
|
||||||
rx: async_channel::Receiver<DownloadProgress>,
|
|
||||||
events: broadcast::Sender<Event>,
|
|
||||||
) {
|
|
||||||
let mut current_total = None;
|
|
||||||
while let Ok(progress) = rx.recv().await {
|
|
||||||
match progress {
|
|
||||||
DownloadProgress::Found { size, .. } => {
|
|
||||||
current_total = Some(size);
|
|
||||||
}
|
|
||||||
DownloadProgress::Progress { offset, .. } => {
|
|
||||||
let _ = events.send(Event::TransferProgress {
|
|
||||||
hash: root.to_hex().to_string(),
|
|
||||||
direction: Direction::Download,
|
|
||||||
bytes_done: offset,
|
|
||||||
bytes_total: current_total,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
DownloadProgress::AllDone(_) | DownloadProgress::Abort(_) => break,
|
|
||||||
_ => {}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Parse a ticket string into (root, format, provider address).
|
/// Parse a ticket string into (root, format, provider address).
|
||||||
pub fn parse_ticket(ticket: &str) -> Result<(Hash, BlobFormat, NodeAddr), String> {
|
pub fn parse_ticket(ticket: &str) -> Result<(Hash, BlobFormat, EndpointAddr), String> {
|
||||||
let ticket: iroh_blobs::ticket::BlobTicket =
|
let ticket: iroh_blobs::ticket::BlobTicket =
|
||||||
ticket.parse().map_err(|e| format!("invalid ticket: {e}"))?;
|
ticket.parse().map_err(|e| format!("invalid ticket: {e}"))?;
|
||||||
Ok((ticket.hash(), ticket.format(), ticket.node_addr().clone()))
|
let (addr, hash, format) = ticket.into_parts();
|
||||||
|
Ok((hash, format, addr))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Load the endpoint secret key from `path`, creating it (mode 0600) on
|
/// Load the endpoint secret key from `path`, creating it (mode 0600) on
|
||||||
@@ -310,7 +255,7 @@ fn load_or_create_secret(path: &Path) -> Result<SecretKey> {
|
|||||||
Ok(hex) => parse_secret(hex.trim())
|
Ok(hex) => parse_secret(hex.trim())
|
||||||
.with_context(|| format!("parsing secret key {}", path.display())),
|
.with_context(|| format!("parsing secret key {}", path.display())),
|
||||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
||||||
let secret = SecretKey::generate(rand::rngs::OsRng);
|
let secret = SecretKey::generate();
|
||||||
let hex = hex_encode(&secret.to_bytes());
|
let hex = hex_encode(&secret.to_bytes());
|
||||||
use std::io::Write;
|
use std::io::Write;
|
||||||
use std::os::unix::fs::OpenOptionsExt;
|
use std::os::unix::fs::OpenOptionsExt;
|
||||||
|
|||||||
@@ -83,12 +83,7 @@ fn wait_complete(socket: &Path, hash: &str, timeout: Duration) -> bool {
|
|||||||
fn forge_ticket(real_ticket: &str, target_hash: &str) -> String {
|
fn forge_ticket(real_ticket: &str, target_hash: &str) -> String {
|
||||||
let ticket = iroh_blobs::ticket::BlobTicket::from_str(real_ticket).unwrap();
|
let ticket = iroh_blobs::ticket::BlobTicket::from_str(real_ticket).unwrap();
|
||||||
let hash = iroh_blobs::Hash::from_str(target_hash).unwrap();
|
let hash = iroh_blobs::Hash::from_str(target_hash).unwrap();
|
||||||
iroh_blobs::ticket::BlobTicket::new(
|
iroh_blobs::ticket::BlobTicket::new(ticket.addr().clone(), hash, iroh_blobs::BlobFormat::Raw)
|
||||||
ticket.node_addr().clone(),
|
|
||||||
hash,
|
|
||||||
iroh_blobs::BlobFormat::Raw,
|
|
||||||
)
|
|
||||||
.unwrap()
|
|
||||||
.to_string()
|
.to_string()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -122,7 +122,11 @@ fn materialize_unknown_hash_is_not_found() {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn gc_drops_unpinned_and_keeps_pinned() {
|
fn gc_drops_unpinned_and_keeps_pinned() {
|
||||||
let daemon = support::spawn_daemon();
|
// Since iroh-blobs 0.103 gc is a periodic loop inside the store, not
|
||||||
|
// an on-demand request; run it fast so the sweep happens in-test.
|
||||||
|
let daemon_dir = tempfile::tempdir().unwrap();
|
||||||
|
let daemon =
|
||||||
|
support::spawn_daemon_with_env(daemon_dir.path(), &[("VARDE_GC_INTERVAL", "1")]);
|
||||||
let mut client = support::Client::connect(&daemon.socket);
|
let mut client = support::Client::connect(&daemon.socket);
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
|
||||||
@@ -143,11 +147,27 @@ fn gc_drops_unpinned_and_keeps_pinned() {
|
|||||||
});
|
});
|
||||||
assert!(reply.ok, "pin failed: {:?}", reply.error);
|
assert!(reply.ok, "pin failed: {:?}", reply.error);
|
||||||
|
|
||||||
|
// The on-demand request reports the new reality honestly.
|
||||||
let reply = client.request(&Request::Gc {});
|
let reply = client.request(&Request::Gc {});
|
||||||
assert!(reply.ok, "gc failed: {:?}", reply.error);
|
assert!(!reply.ok, "on-demand gc should be unimplemented");
|
||||||
match reply.data {
|
|
||||||
Some(ResponseData::GcDone { blobs_removed }) => assert!(blobs_removed >= 1),
|
// Wait for the periodic sweep to collect the loose blob.
|
||||||
other => panic!("unexpected gc reply: {other:?}"),
|
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20);
|
||||||
|
loop {
|
||||||
|
let err = materialize(
|
||||||
|
&mut client,
|
||||||
|
&doomed,
|
||||||
|
&dir.path().join("probe"),
|
||||||
|
MaterializeMode::Copy,
|
||||||
|
);
|
||||||
|
if err.is_err() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
std::time::Instant::now() < deadline,
|
||||||
|
"unpinned blob survived gc for 20s"
|
||||||
|
);
|
||||||
|
std::thread::sleep(std::time::Duration::from_millis(500));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Pinned tree still materializes; loose blob is gone.
|
// Pinned tree still materializes; loose blob is gone.
|
||||||
@@ -167,21 +187,29 @@ fn gc_drops_unpinned_and_keeps_pinned() {
|
|||||||
.unwrap_err();
|
.unwrap_err();
|
||||||
assert_eq!(err, ErrorCode::NotFound);
|
assert_eq!(err, ErrorCode::NotFound);
|
||||||
|
|
||||||
// Unpinning and collecting again drops the tree too.
|
// Unpinning lets the next sweep drop the tree too.
|
||||||
let reply = client.request(&Request::Unpin {
|
let reply = client.request(&Request::Unpin {
|
||||||
hash: kept_root.clone(),
|
hash: kept_root.clone(),
|
||||||
});
|
});
|
||||||
assert!(reply.ok);
|
assert!(reply.ok);
|
||||||
let reply = client.request(&Request::Gc {});
|
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20);
|
||||||
assert!(reply.ok);
|
loop {
|
||||||
let err = materialize(
|
let err = materialize(
|
||||||
&mut client,
|
&mut client,
|
||||||
&kept_root,
|
&kept_root,
|
||||||
&dir.path().join("out3"),
|
&dir.path().join("out3"),
|
||||||
MaterializeMode::Copy,
|
MaterializeMode::Copy,
|
||||||
)
|
);
|
||||||
.unwrap_err();
|
if let Err(code) = err {
|
||||||
assert_eq!(err, ErrorCode::NotFound);
|
assert_eq!(code, ErrorCode::NotFound);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
std::time::Instant::now() < deadline,
|
||||||
|
"unpinned tree survived gc for 20s"
|
||||||
|
);
|
||||||
|
std::thread::sleep(std::time::Duration::from_millis(500));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
Reference in New Issue
Block a user