Skip to content

Latest commit

 

History

History
159 lines (140 loc) · 11.3 KB

requirements.md

File metadata and controls

159 lines (140 loc) · 11.3 KB

Requirements and choosing a deploy path

What a deCDN node needs from its host and network, and which of this repo's four deploy paths fits. It applies to a node operator's cache node and a publisher's origin node alike: an origin is the same node with an origin backend (node-operators.md, publishers.md). Protocol facts (bond sizing, fees) are not here: they come from the deCDN ADRs.

Choosing a path

Ansible cloud-init Docker Compose Helm
Target VMs, bare metal one cloud VM, no control machine one host running Docker Kubernetes
Host hardening (firewall, SSH, patching) yes, baseline yes, baseline no, yours no, the cluster's
Fleets yes, one inventory one user-data per VM one host per compose project one release per node
Secrets host file, git-ignored inventory, or ansible-vault in a private inventory repo host file, written over SSH host file operator-created Secrets
Install source signed release tarball, build on the host from a git ref, or local build signed release tarball image by digest (enforced) image by digest (recommended) or tag
Chain config decdn_network profile decdn_network profile decdn config init --chain explicit values
Monitoring opt-in Grafana Cloud agent opt-in Grafana Cloud agent (token in a host file) opt-in Grafana Cloud agent (alloy profile, token in a host file) ServiceMonitor, PrometheusRule, dashboards
Backup / decommission make backup / make decommission the Ansible targets, from a workstation inventory decdn-compose backup / down PVC snapshot
Guide ansible/README.md cloud-init/README.md compose/README.md charts/decdn-node/README.md

If you are unsure, start with Docker Compose: one host, one wrapper, no control machine. It installs the node but leaves the host's firewall, SSH and patching to you. For a fleet, or a VPS or dedicated server you want hardened from scratch, use Ansible, or cloud-init for a single VM when you would rather not run Ansible from a workstation. They are the only two that harden the host as well as installing the node, and they run the same roles.

Platforms

Supported How it is tested
OS Debian 12 (bookworm), Debian 13 (trixie), Ubuntu 24.04 (noble), Ubuntu 26.04 (resolute) molecule converges decdn_node on all four in systemd containers; grafana_alloy's install path on Debian 12 only (its disabled path on all four); sponsord and sponsord_onramp on Debian 12 and Ubuntu 24.04; baseline on real hosts
Architecture x86_64, aarch64 upstream builds both; the Ansible role derives the target from the host
Ansible (control machine) ansible-core ≥ 2.15 CI runs the current release
cloud-init the provider image's own; the bootstrap installs its pinned ansible-core on the host the node template is booted with the distro's cloud-init in Debian 12 and Ubuntu 26.04 containers, to a running node; the sponsor template in Ubuntu 26.04, to running sponsord, sponsord-onramp and nginx
Kubernetes ≥ 1.25 rendered and validated with kubeconform against 1.30
Docker Compose v2 with env_file.required and depends_on.restart support (2.24+), with systemd-journald (or a compose.override.yaml setting another logging driver); Python ≥ 3.11 for the decdn-compose wrapper rendered in CI with every profile on; the wrapper's decisions unit-tested

On Ubuntu 25.10 and later, sudo is sudo-rs; see the ansible_become_exe note in ansible/inventory/hosts-node.yml.example.

Network

  • Inbound: udp/4433 (QUIC) from anywhere. This is the node's only public port. Open it in the host firewall (the Ansible baseline does this) and in your cloud provider's security group or firewall, which the host cannot see.
  • A sponsord-onramp host also needs inbound tcp/80 and tcp/443 for nginx, from anywhere: 443 serves the onramp, 80 answers lego's ACME HTTP-01 challenge and redirects to https. On Ansible, baseline opens both for hosts in sponsord_onramp_hosts while sponsord_onramp_proxy is nginx (the default; none opens neither), and nginx comes from the distribution. On Compose the onramp stays on loopback behind your own TLS proxy, which opens its own ports. Either way, open them in the provider's firewall too, and point the domain's A/AAAA records at the host before the first deploy.
  • An iroh relay host (Ansible, or Compose's relay profile) needs inbound tcp/80, tcp/443 and udp/7842 from anywhere: 443 carries the relay and Let's Encrypt's TLS-ALPN-01 challenge, 80 iroh's captive-portal probe, and 7842 QUIC address discovery (unless iroh_relay_enable_quic_addr_discovery is false). On Ansible, baseline opens them for hosts in iroh_relay_hosts; on Compose you open them in the host firewall. Either way, open them in the provider's firewall too, and point the relay hostname's A/AAAA records at the host. It also needs outbound HTTPS to Let's Encrypt. A relay cannot share a host with sponsord-onramp (both want 80 and 443). It carries the traffic of every peer pair it relays, so size its bandwidth for that, and set rate limits (iroh_relay_client_rx_bytes_per_second) on a metered link. By default only the inventory's deCDN nodes and the IDs in iroh_relay_allowlist may relay through it: its deploy reads each node's ID on the node, so it needs SSH to every node and every node deployed first (on Compose you list the IDs in its config yourself). decdn clients cannot be listed (a fresh key per fetch), so a node behind NAT that serves clients needs relays with iroh_relay_access: everyone.
  • An iroh DNS server host (Ansible, or Compose's dns profile) needs inbound tcp/443, udp/53 and tcp/53 from anywhere: 443 carries nodes' record PUTs and Let's Encrypt's TLS-ALPN-01 challenge, 53 answers resolvers (tcp for answers too large for udp). On Ansible, baseline opens them for hosts in iroh_dns_server_hosts; on Compose you open them in the host firewall. Either way, open them in the provider's firewall too. It also needs a delegated zone: at the parent zone, an NS record for the server's hostname naming that hostname, and a glue A record with the host's public address. It needs outbound HTTPS to Let's Encrypt. Port 53 is bound on one address (the host's default IPv4), so systemd-resolved's stub listener can stay. It cannot share a host with an iroh relay or sponsord-onramp (all want 443). See roles/iroh_dns_server.
  • A public gateway host needs inbound tcp/80 and tcp/443 for its TLS proxy, from anywhere: 443 serves the gateway, 80 redirects to https. On Ansible the proxy is the role's nginx, and baseline opens both for hosts in decdn_gateway_hosts; on Compose's gateway profile it is your own, and you open them in the host firewall. Either way, open them in the provider's firewall too. Point both <domain> and *.<domain> at the host. Its certificate comes over ACME DNS-01, so it needs outbound HTTPS to Let's Encrypt and to your DNS provider's API, plus the RPC endpoint and QUIC/UDP to the nodes it pays. It cannot share a host with sponsord-onramp, an iroh relay or an iroh DNS server (all want 443). nginx comes from the distribution. See roles/decdn_gateway.
  • Outbound: HTTPS to your RPC provider and to any cache origin; QUIC/UDP to peers; HTTPS to the iroh relays (n0's public relays unless decdn_relay_urls names your own; see roles/iroh_relay).
  • NAT: a node without direct inbound reachability still works through iroh relays, at a latency cost. A node that serves paid traffic should be directly reachable, and the multiaddr it registers on-chain must be its public address (/ip4/<public-ip>/udp/4433/quic-v1).
  • IPv6: supported alongside IPv4. Since decdn/decdn#2144 (869141e9) the node's QUIC listener binds both 0.0.0.0:4433 and [::]:4433, and a dual-stack node registers both its /ip4/ and /ip6/ multiaddrs; older builds are IPv4-only on 4433. On Ansible, baseline_preserve_ipv6_autoconf (default true) keeps SLAAC addresses alive under DevSec hardening; set it false on static-IPv6 hosts. Multi-homed hosts may need baseline_rp_filter_loose: true.
  • Loopback only, never exposed: metrics 127.0.0.1:9090, admin RPC 127.0.0.1:9191; on a gateway host, the gateway 127.0.0.1:8484 and its metrics 127.0.0.1:9094. Reach them over SSH, or ship metrics with the monitoring options.
  • Clock: keep NTP running (baseline installs chrony). decdn setup's pre-flight checks clock skew.
  • Bandwidth: the node serves what it is asked for. On metered bandwidth, cap egress with [load_shed] egress_budget_mbps in node.toml (decdn_load_shed_egress_budget_mbps on Ansible, config.load_shed.egress_budget_mbps on Helm), and set a billing alert with your provider.

Disk

Everything lives on the data volume (/var/lib/decdn, or the PVC):

cache_size_mb                    the cache (role default 10 GiB)
+ disk_headroom_mb               free space the daemon keeps on the volume (default 8 GiB)
+ receipts                       rotated download-receipt log, bounded by its [receipts] settings
+ daemon state (redb stores)     small, grows with activity; no published bound
+ an fs origin, if the node holds one

max_blob_size_mb must be ≤ cache_size_mb: it caps the bytes the node holds for one blob. A blob from the node's own origin above it is refused (BlobTooLarge); a pull-through serves ranges of a larger blob while the bytes held for it fit, and stops once they do not. Size it to the largest single object you expect to serve whole (role default 1 GiB). Put /var/lib/decdn on the fast disk.

The Ansible source install method adds, outside the data volume, a Rust toolchain and a git clone per role under /var/lib/decdn-build, plus, while it builds, a work directory with the whole build tree (several GB, removed after a successful install). Every new commit is a full build from scratch, crates downloaded again.

CPU and memory

Upstream publishes no minimum figures yet, so this repo quotes none. Watch the node's own metrics (decdn node top, the dashboards in monitoring/) under real load and size from that.

The Ansible source install method compiles on the host, which needs far more memory and CPU than running the daemon, for the length of the build (this repo has not measured a real build yet). On a small host, lower *_source_build_jobs, or build elsewhere and use manual.

Accounts and keys

  • An RPC endpoint for the chain (Arbitrum Sepolia today). The public endpoint that config init writes works for light use; production nodes should use their own provider. Keep the URL secret if it embeds an API key.
  • An operator wallet: the eth keystore decdn key-gen creates, funded for gas and the bond before decdn setup. The bond comes from the on-chain bondRequired(mbps) curve (ADR 026).
  • Optional: a Grafana Cloud stack (Ansible grafana_alloy, Compose alloy), and an age key pair for encrypted backups (docs/lifecycle.md).