Customer app edge deployment (app.*)

Last Updated: 2026-06-12 (blockstore v1 vs multi-region triangle)

Deploy **apps/file-explorer** and **substratum-gateway** behind the **app.*** origin (production deployment, OAuth and PDS origins).

Topology

Component	Role
**infra/app Pulumi stack**	Droplet, reserved IP, firewall, DNS A for app.*
Caddy (cloud-init)	TLS, SPA from /var/www/app, proxy /api + /.well-known → gateway
**substratum-gateway**	Internal :18080 — customer API, OAuth, sessions
Managed Postgres	Same database as ops-api and PDS proxy (DATABASE_URL / substratum_gateway)
**apps/file-explorer build**	Vite static output rsync'd to /var/www/app

One browser origin: PUBLIC_BASE_URL = https://app.<domain> for cookie, OAuth redirect, and client metadata.

URL map (prod)

Two different URL families — do not conflate them.

Purpose	Example	Where it comes from
Public browser origin	https://app.substratum.cloud	pulumi stack output appPublicBaseUrl (or https://$(pulumi stack output appPublicDomain)) — set **PUBLIC_BASE_URL** on gateway
Caddy upstream (internal)	http://127.0.0.1:18080	**gatewayUpstream** / Spindle APP_GATEWAY_UPSTREAM — where Caddy proxies /api and /.well-known

Garage v1 runs gateway on the same droplet as Caddy, so the upstream is loopback — not the public HTTPS URL. Putting https://app.substratum.cloud in APP_GATEWAY_UPSTREAM would proxy-loop through Caddy.

Spindle / CI: optional APP_DOMAIN (default app.substratum.cloud). You usually do not set APP_GATEWAY_UPSTREAM unless gateway listens elsewhere (private IP + port).

Prerequisites

• Pulumi stack **data** provisioned — see data deployment (DATABASE_URL + bootstrap)
• Pulumi stack **app** provisioned — see [infra/app/AGENTS.md](../../infra/app/AGENTS.md)
• S3-compatible blockstore (or FlatFS for small installs) configured on gateway
• Production secrets: JWT_SECRET, SWARM_MASTER_SECRET, LIBP2P_IDENTITY_KEY — see Generate gateway secrets below

Generate gateway secrets (first time)

Run once from the repo root before configuring Spindle or pulumi config set gateway* --secret on infra/app. Store outputs in a password manager; do not commit.

# Session JWT (Spindle: APP_JWT_SECRET)
openssl rand -base64 32

# Swarm PSK root (Spindle: APP_SWARM_MASTER_SECRET)
openssl rand -base64 32

# Stable libp2p identity — run once; never rotate on deploy (Spindle: APP_LIBP2P_IDENTITY_KEY)
cargo run -p substratum-ingress --bin generate_keys

generate_keys prints PeerId: … and LIBP2P_IDENTITY_KEY=<base64> (protobuf Ed25519 keypair). Use the LIBP2P_IDENTITY_KEY= line value only. First compile may take several minutes.

Alternative: set the same values on the Pulumi stack instead of Spindle:

cd infra/app
pulumi config set gatewayJwtSecret "$(openssl rand -base64 32)" --secret
pulumi config set gatewaySwarmMasterSecret "$(openssl rand -base64 32)" --secret
pulumi config set gatewayLibp2pIdentityKey "$(cargo run -p substratum-ingress --bin generate_keys 2>/dev/null | sed -n 's/^LIBP2P_IDENTITY_KEY=//p')" --secret

1. Provision infrastructure

cd infra/app
npm ci
pulumi stack select app
pulumi config set digitalocean:token --secret
pulumi config set domain app.substratum.cloud
pulumi config set gatewayUpstream http://127.0.0.1:18080
pulumi config set doProjectId cd1eca0e-c563-4501-9896-0c21db8b3b55
pulumi up

If gateway runs on the same droplet, use a larger size:

pulumi config set size s-2vcpu-4gb
pulumi up

2. Gateway on the host

CI: Spindle app.yml runs [deploy-app-gateway.sh](../../scripts/ci/deploy-app-gateway.sh) after pulumi up — builds the release binary, installs [substratum-gateway.service](../../infra/app/substratum-gateway.service), and polls GET /api/health on loopback.

Manual / local parity:

export PULUMI_ACCESS_TOKEN=… DIGITALOCEAN_TOKEN=…
# plus gateway secrets (see CI secrets table below)
./scripts/ci/deploy-app-gateway.sh

Example env (systemd unit uses /etc/substratum/gateway.env):

Variable	Value
HOST	127.0.0.1
PORT	18080
PUBLIC_BASE_URL	https://app.substratum.cloud
DATABASE_URL	From [infra/data](../../infra/data/AGENTS.md) — substratum_gateway role after bootstrap
JWT_SECRET	Production secret
SWARM_MASTER_SECRET	Production secret
LIBP2P_IDENTITY_KEY	Stable base64 keypair
CORS_ALLOWED_ORIGINS	https://app.substratum.cloud (same origin when proxied)

Verify internal health:

curl -sS http://127.0.0.1:18080/api/health

3. Build and rsync file-explorer SPA

From repo root:

pnpm install
pnpm nx build file-explorer

DROPLET_IP="$(cd infra/app && npx pulumi stack output dropletIp)"
KEY_FILE="$(mktemp)"
cd infra/app && npx pulumi stack output appSshPrivateKey --show-secrets > "${KEY_FILE}"
chmod 600 "${KEY_FILE}"

ssh -i "${KEY_FILE}" -o StrictHostKeyChecking=accept-new root@"${DROPLET_IP}" \
  'cloud-init status --wait && systemctl is-active --quiet caddy'

rsync -avz --delete \
  -e "ssh -i ${KEY_FILE} -o StrictHostKeyChecking=accept-new" \
  dist/apps/file-explorer/ \
  "root@${DROPLET_IP}:/var/www/app/"

ssh -i "${KEY_FILE}" root@"${DROPLET_IP}" \
  'chown -R caddy:caddy /var/www/app && systemctl reload caddy'

rm -f "${KEY_FILE}"

4. Smoke tests

Check	Expected
SPA	curl -sI https://app.substratum.cloud/ → 200
OAuth metadata	curl -sS https://app.substratum.cloud/.well-known/oauth-client-metadata.json → JSON; client_id matches URL
API	curl -sS https://app.substratum.cloud/api/health → {"status":"ok"}
Login	Browser OAuth → substratum_session on app.* origin

CI (Spindle)

.tangled/workflows/app.yml — on main push (or manual):

1. **deploy-app.sh** — pulumi refresh + pulumi up (app stack) + **deploy-app-gateway.sh**
2. pnpm nx build file-explorer
3. rsync to droplet + reload Caddy via scripts/ci/deploy-edge.sh
4. **verify-app-live.sh** — loopback + public GET /api/health (fails CI on 502)

Local operator parity: same scripts from repo root (see scripts/ci/AGENTS.md).

Spindle secrets

Secret	Required	Notes
PULUMI_ACCESS_TOKEN	yes	Pulumi Cloud
DIGITALOCEAN_TOKEN	yes	pulumi up on app stack
APP_JWT_SECRET	yes*	Session JWT — openssl rand; or gatewayJwtSecret stack config
APP_SWARM_MASTER_SECRET	yes*	Swarm PSK root — openssl rand; or gatewaySwarmMasterSecret stack config
APP_LIBP2P_IDENTITY_KEY	yes*	Stable base64 libp2p keypair — generate_keys; or gatewayLibp2pIdentityKey stack config
APP_DOMAIN	no	Default app.substratum.cloud
APP_SSH_HOST	no	Droplet IP override
APP_GATEWAY_UPSTREAM	no	Default http://127.0.0.1:18080
APP_PDS_PUBLIC_URL	no	Default https://pds.substratum.cloud
APP_S3_* / SPACES_*	no	Blockstore — see table below; omit for FlatFS (not recommended in prod)

At least one source per row (Spindle env or matching Pulumi stack secret).

Blockstore (S3 / DO Spaces)

CI writes S3_* into /etc/substratum/gateway.env. Resolution order: APP_S3_* → SPACES_* (same keys as marketing Spindle) → defaults.

Your Spindle secret	Gateway env (on droplet)	Notes
SPACES_ACCESS_KEY_ID	S3_ACCESS_KEY	Or set APP_S3_ACCESS_KEY
SPACES_SECRET_ACCESS_KEY	S3_SECRET_KEY	Or set APP_S3_SECRET_KEY
SPACES_BUCKET	S3_BUCKET	Or set APP_S3_BUCKET — use a dedicated blocks bucket if marketing uses a downloads bucket
(not in Spindle)	S3_ENDPOINT	Default https://nyc3.digitaloceanspaces.com; override with SPACES_ENDPOINT or APP_S3_ENDPOINT
(not in Spindle)	S3_REGION	Default nyc3; override with SPACES_REGION or APP_S3_REGION if bucket is in another region

Garage v1: Reusing the marketing SPACES_* secrets in Spindle is fine — one gateway on the app droplet, one DO Spaces bucket/region. No extra blockstore keys unless the bucket is not in nyc3 (add SPACES_REGION) or blocks should not share the marketing bucket (set APP_S3_BUCKET=substratum-blocks).

Post–Garage v1 (target): The Global Triangle swarm must exist on three cloud nodes — three separate substratum-gateway processes, each with its own stable LIBP2P_IDENTITY_KEY / PeerId, running libp2p in the same private mesh (shared SWARM_MASTER_SECRET, per-DID PSKs). They dial each other via BOOTSTRAP_NODES and replicate uploads via PINNING_TARGETS (ADR 03, ADR 17). Place them across availability zones / continents for resilience.

That is not three clones of the app.* droplet with one identity, and not covered by this runbook. Garage v1 ships one mesh participant (loopback gateway on the app droplet). A future runbook must cover per-node deploy, mesh wiring, and blockstore strategy (regional buckets vs mesh-hot storage — TBD).

Troubleshooting

Symptom	Likely cause
502 on /api/…	Gateway not running or wrong gatewayUpstream
OAuth metadata 404	Caddy not proxying /.well-known or gateway down
Cookie not sent	Split origins — UI and API must share app.*
RLS errors	DATABASE_URL uses superuser instead of substratum_gateway
kex_exchange_identification: Connection reset by peer or Connection closed during rsync	Droplet still booting / cloud-init running (Caddy install), or stale APP_SSH_HOST
Permission denied (publickey) on rsync	Deploy key mismatch

Related

• Production deployment
• Garage v1 rollout
• [infra/app/AGENTS.md](../../infra/app/AGENTS.md)