Garage v1 rollout — PDS, entitlements, and support
Last Updated: 2026-06-10 (Phase 3 proxy tests; Phase 4 ops API + admin scaffold)
Operational rollout plan for Garage (0–500 users): Substratum-operated PDS (pds.substratum.cloud), entitlement enforcement, manual billing, and operator support tooling. Design rationale lives in ADR 37 and ADR 38; this runbook is the implementation and launch sequence.
Audience
- Engineering — build order, exit criteria, parallel tracks.
- Operators — launch gates, invite-only posture, day-one support flow (grant → Discourse → customer signup).
Scope
| In Garage v1 | Out of scope (post-v1) |
|---|---|
| Entitlement core + manual billing adapter | Stripe / regional PSP webhooks |
| Gateway + sync-worker entitlement gates | Marketing checkout automation |
| PDS repo authz proxy on operated PDS | Pending entitlement by email (pre-signup) |
| Invite-only Substratum login | Open self-service PDS signup |
Operator admin API + admin UI at admin.* (6 locales) | Platform cross-tenant DLQ console |
Discourse at support.* (invite/revoke on grant/lapse) | Machine-credential rotation automation |
| Staff company SSO via Substratum PDS OAuth | Third-party OIDC (Google/Okta) for staff |
Product promises: Business model — Substratum Once, base membership, Discourse for paid login customers.
Topology at launch
| Host | Role | Notes |
|---|---|---|
substratum.cloud | Marketing | Existing marketing runbook; not PDS |
pds.substratum.cloud | Customer + staff identity | Block volume + reserved IP; proxy in front of repo mutations |
app.* | Gateway + customer SPA | Entitlement checks at upload + sync boundaries |
admin.* | Operator SPA | e.g. admin.substratum.cloud; separate origin; substratum_ops_session; Lingui |
support.* | Customer support (Discourse) | Discourse UI; OIDC client to bridge |
id.support.* | OIDC bridge issuer | AT Proto OAuth upstream; entitlement gate (ADR 39) |
Hard gates
Do not skip these — they are normative in ADR 37 C10 and ADR 38.
| Gate | Rule |
|---|---|
| No open PDS signup without proxy | General Substratum PDS registration stays invite-only until repo authz proxy (Phase 3) is live in production |
| No raw entitlement SQL | All plan/capability changes go through manual billing adapter → audit log |
| Customer session ≠ operator session | substratum_session must not authorize admin routes; staff use PDS OAuth → substratum_ops_session + operator_role row |
| Garage v1 needs ops UI | Runbook supplements the UI; they do not replace ADR 38 step D — no operator CLI |
| Discourse server-side only | Invite/revoke from adapter; no Discourse admin API keys in browser |
Implementation spine
Four phases share one entitlement service in Postgres. Build in order; parallel work is allowed where noted.
Phase 1 — Entitlement core
Goal: Single source of truth for capabilities; operators can grant/lapse without Stripe.
Deliverables
| Item | ADR | Implementation notes |
|---|---|---|
Capability columns on entitlement (metadata_write_allowed, safety_net_allowed, lapse/grace, PSP ids) | 37 R1–R2, R15 | Extend existing plan / entitlement tables (ADR 32) |
EntitlementPolicyPort | 37 §4, NR6 | Shared port for gateway, PDS proxy, admin API — wrap or extend PostgresUploadPolicy in crates/ingress |
entitlement_audit_log schema | 38 R5 | Append-only; operator_did or machine principal |
operator_role registry | 38 R18 | staff_did → role(s): support_read, entitlement_mutator, billing_reconciler |
| Manual billing adapter module | 37 R3–R4, 38 R3–R4 | Normalized events: manual.grant, checkout.completed, lapse, grace extend — invoked by operator admin API |
| Entitlement admin runbook (procedures) | 38 R6 | Day-to-day flows via admin.* UI + admin API (not CLI) |
Exit criteria
- [x] Manual adapter grant Once sets capabilities per ADR 37 CC7 (integration tests).
- [x] Every mutation writes audit row with required
reason/ticket_ref. - [x] Idempotent replay of same adapter event does not corrupt state.
- [x]
EntitlementPolicyPortanswerseffectiveCapabilities(did)for tests.
Verification scenarios
| Scenario | Expected |
|---|---|
manual.grant Once | metadata_write_allowed=true, safety_net_allowed=false |
| Full lapse (no Once) | Both capabilities false |
| Adapter event without ticket ref | Rejected (400) |
Phase 2 — Gateway enforcement
Goal: Close bypass paths through sync workers and safety-net uploads before operated PDS accepts customers.
Deliverables
| Item | ADR | Implementation notes |
|---|---|---|
Real GET /api/v1/me/limits | 37 R11 | Replace stub in crates/ingress/src/router/handlers/session.rs |
safety_net_allowed on SaaS upload reserve | 37 R9, R12 | Existing quota reserve/commit in upload_policy.rs |
| Outbox enqueue gate | 37 R9–R10 | receipt_sync_outbox / catalog_sync_outbox when target is Substratum PDS |
| Worker pre-write gate | 37 R9 | Before putRecord in receipt/catalog workers |
| Non-retryable entitlement denial | 37 CC6 | Surface on customer /account/sync-failures (ADR 35) |
| Home gateway entitlement URL | 37 §4 | ENTITLEMENT_SERVICE_URL when pds_url is Substratum PDS; stale cache → pause sync, not block local home copy |
Exit criteria
- [x] Lapsed DID: no new Substratum PDS sync enqueued; worker fails closed on pre-write.
- [x] Self-hosted gateway: local catalog/blockstore without Substratum PDS sync still works offline.
- [x]
/me/limitsreturns capability flags + CTA URLs for account UI.
Verification scenarios
| Scenario | Expected |
|---|---|
| Base lapsed, no Once, catalog write → Substratum PDS | HTTP 403 at enqueue |
| Home gateway, lapsed, local-only upload | Succeeds without cloud entitlement call |
| Active base, safety-net upload over quota | HTTP 413 at reserve |
Still blocked after Phase 2: Open customer signup on pds.substratum.cloud (direct PDS XRPC bypasses gateway).
Phase 3 — Operated PDS + authz proxy
Goal: Safe to host customer repos; staff DIDs for company SSO.
Deliverables
| Item | ADR | Implementation notes |
|---|---|---|
infra/pds Pulumi stack | ADR 41 | Stateless droplet + Spaces blobstore + reserved IP; separate from marketing droplet |
| Tranquil PDS upstream | — | Repo in Postgres tranquil_pds; blobs in Spaces; pds.substratum.cloud canonical URL |
| Repo authz proxy (Caddy or sidecar) | 37 R5–R8 | Route only com.atproto.repo.{create,put,delete}Record for gated collections |
| Proxy entitlement lookup | 37 CC1 | Same Postgres / internal API as gateway |
deleteRecord when lapsed | 37 R7, C5 | Allow for cloud.substratum.* cleanup and BYO migration |
| Staff handles on PDS | 38 R17–R18 | Provision via runbook; insert operator_role rows |
| Invite-only customer signup | 37 R16 | POST /api/v1/pds/signup gated on entitlement or invite |
Proxy routing (normative)
| XRPC | cloud.substratum.* | Check |
|---|---|---|
createRecord / putRecord | yes | metadata_write_allowed |
deleteRecord | yes | Valid repo owner/session — even when lapsed |
| Reads / other collections | — | Pass through |
Deny putRecord with substratum.entitlement.metadata_write_denied (ADR 37 R18).
Exit criteria
- [x] Direct
putRecordto Substratum PDS respects capabilities (not just gateway path) —apps/pds-authz-proxy+gate_httpintegration tests. - [x] Lapsed customer can
deleteRecordon Substratum lexicon collections — proxy gate + integration tests. - [ ] At least one staff DID can complete PDS OAuth (prep for Phase 4).
- [ ] Staging smoke: grant → signup → OAuth → receipt sync → proxy allow.
Code complete (2026-06-10): apps/pds-authz-proxy (HTTP integration tests), Compose pds-upstream + pds-authz-proxy. Staging/production pending: Spindle pds.yml provisions infra/pds; operator still deploys PDS upstream + authz proxy binary on the droplet — see pds-deployment.md.
Verification scenarios
| Scenario | Expected |
|---|---|
Active Once, putRecord receipt via proxy | Allow |
Lapsed, no Once, putRecord | Proxy 403 |
Lapsed, no Once, deleteRecord | Proxy allow |
| BYO handle on external PDS | Proxy N/A |
See also OAuth and PDS origins for same-origin and issuer metadata when wiring app.* and admin.*.
Phase 4 — Support + operator UI
Goal: Operators run Garage revenue and support without SQL; customers get Discourse.
Maps to ADR 38 steps B–D. Garage v1 is not complete until step D ships.
Deliverables
| Step | Item | Notes |
|---|---|---|
| B | Ops OAuth routes + substratum_ops_session | Reuse crates/auth; separate OAuth client / redirect for admin.* |
| B | Operator admin API | /internal/v1/ops/… — lookup by did/handle/email, write mutations, audit tail, sync-failures |
| C | Discourse + OIDC bridge | Deploy Discourse at support.*, bridge at id.support.*, enable bundled OpenID Connect; entitlement-gated login; lapse suspend via Discourse Admin API |
| C | File-explorer boundary | /me/limits exposes support_seat_allowed + support_url; /account external link to support.* only — no in-app Discourse or bridge routes (ADR 39 R13) |
| D | apps/admin at admin.* | Isolated Vite app; Lingui + 6 locales; reuse @substratum/ui-kit; app-local operator pages — no imports from file-explorer (ADR 33 §8) |
Operator auth (v1)
| Surface | Mechanism |
|---|---|
| Operator UI | AT Protocol OAuth → pds.substratum.cloud → substratum_ops_session |
| Admin API (browser) | Same ops session + operator_role check |
| Automation | Scoped machine credential to admin API; distinct audit principal |
Provisioning: create staff handle on Substratum PDS → insert operator_role row → staff signs in at admin.*.
Exit criteria
- [ ] Staff OAuth → grant Once with ticket ref → capabilities + audit + Discourse invite queued.
- [ ] Customer DID without role row gets 403 on admin API after OAuth.
- [ ] Read-only role cannot call grant endpoints (403).
- [ ] Entitled customer opens
support.*in browser (not via file-explorer SSO); Discourse OIDC + bridge login succeeds. - [x] File-explorer
/accountlinks tosupport.*; SPA contains no OIDC bridge or Discourse embed routes. - [ ] Admin UI
LanguagePickerswitches all chrome without reload (ADR 14).
In progress (2026-06-10): Ops OAuth + /internal/v1/ops/… admin API (ADR 38-B); apps/admin scaffold (Login, Lookup, Grant Once). Not started: Discourse + OIDC bridge (step C).
Customer support flow (launch day)
Parallel workstreams
| Track | Start after | Can proceed in parallel with |
|---|---|---|
| Entitlement schema + adapter | Now | — |
| Gateway / worker gates | Phase 1 port | PDS infra design |
| PDS infra + proxy | Phase 1 port | Phase 2, admin API design |
| Admin API + ops OAuth | Phase 1 adapter + operator_role | Phase 3 (needs staff DIDs before UI login) |
Admin UI at admin.* | Admin read API (mock OK early) | Discourse adapter |
| Discourse adapter | Manual billing adapter events | Admin UI |
Suggested team split:
- Backend: Phase 1 → 2 → 38-B while infra builds Phase 3.
- Infra: Phase 3 stack + proxy.
- Frontend:
apps/adminafter read API contract; all six locale catalogs per string change.
Launch checklist (invite-only Garage)
Complete all four phases in staging before production invite-only launch.
Pre-launch (staging)
- [ ] Phase 1–4 exit criteria green in staging.
- [ ] PDS proxy on staging
pds.*matches production routing table. - [ ] Staff DIDs provisioned;
admin.*reachable on staging origin. - [ ] Discourse at
support.*+ OIDC bridge atid.support.*; entitled DID login smoke; lapse deactivate smoke. - [ ] Runbook: grant Once, grant base, lapse, extend grace, staff onboarding.
- [ ] Customer export/delete path documented when lapsed (ADR 29).
Production cutover
- [ ] Deploy entitlement migrations to production Postgres.
- [ ] Deploy gateway with Phase 2 gates enabled.
- [ ] Deploy PDS stack + proxy; DNS
pds.substratum.cloud→ reserved IP. - [ ] Deploy Discourse at
support.*and OIDC bridge atid.support.*(TLS, backups per ops runbook). - [ ] Deploy
apps/adminatadmin.*+ internal admin routes (VPN or allowlist per ADR 38 C8). - [ ] Confirm
POST /api/v1/pds/signupis invite/entitlement gated — not open registration. - [ ] Marketing site unchanged; checkout remains manual/ops-driven until PSP adapter ships.
Post-launch smoke
- [ ] Operator grant → customer signup → OAuth → upload → receipt on Substratum PDS.
- [ ] Operator lapse → customer
putRecorddenied;deleteRecordallowed. - [ ]
/me/limitsand sync-failures reflect lapse for customer. - [ ] Audit log shows
operator_didfor grant.
Current implementation status
As of this runbook (2026-06-10). Update this section when phases land.
| Component | Status |
|---|---|
EntitlementPolicyPort + PostgresEntitlementPolicy | Shipped — crates/entitlement |
Manual billing adapter (crates/entitlement) | Shipped — wired to operator admin API |
entitlement_audit_log / operator_role | Shipped — migration m20260609_000001 |
Capability columns on entitlement | Shipped |
| Entitlement admin runbook | entitlement-admin.md |
GET /api/v1/me/limits | Shipped — capability flags, CTA URLs, support_seat_allowed + support_url |
Sync-failures API (GET/POST /api/v1/me/sync-failures) | Shipped — receipt_sync failed jobs + retry |
Account / Explorer lapse banners + /account/sync-failures | Shipped — file-explorer; /account external link to support.* |
PostgresUploadPolicy (plan/quota) | Shipped — safety_net_allowed gate on upload reserve |
| PDS repo authz proxy | Code complete — apps/pds-authz-proxy + gate_http tests; Compose + pds-deployment.md |
infra/data | Pulumi stack ready — DO Managed Postgres; bootstrap per data-deployment.md |
infra/pds | Pulumi stack ready — droplet deploy + smoke manual |
infra/admin | Pulumi stack ready — admin.* edge (Caddy SPA + ops-api proxy); deploy per admin-deployment.md |
infra/app | Pulumi stack + CI gateway deploy — app.* edge (Caddy SPA + substratum-gateway); see app-deployment.md |
| Operator admin API / ops OAuth | Shipped — apps/ops-api (:18280); not on customer gateway (:18080) |
apps/admin at admin.* | Scaffold — Login, Lookup, Grant Once; infra + rsync via infra/admin |
Discourse + OIDC bridge (support.*, id.support.*) | Not implemented |
| Stripe / PSP webhooks | Deferred |
Post-v1 (explicit deferrals)
| Item | ADR | Trigger |
|---|---|---|
| Pending entitlement by email | 38 E | Automated pre-signup checkout |
| Stripe adapter | 37 R16, 32 deferred | Manual ops path stable |
| Marketing checkout → billing | 36 + 37 | PSP chosen per region |
| Machine credential rotation | 38 F | CLI automation in production |
| Open PDS self-signup | 37 C10 | Proxy proven + abuse policy |