ADR 38: Support and Entitlement Admin Tooling
Status: Proposed
Date: 2026-06-09
Last Updated: 2026-06-10 (Discourse at support.*; see ADR 39 OIDC bridge)
Terms (this ADR)
| ID | Term | Meaning |
|---|---|---|
| Rn | Functional requirement | Numbered obligation (R1–R7a, R8–R18 in this ADR). |
| NRn | Non-functional requirement | Quality attribute (NR1–NR7). |
| Cn | Constraint | Non-negotiable boundary (C1–C10). |
| CCn | Cross-cutting challenge | Risk spanning components (CC1–CC7). |
| Operator | Substratum staff | Trusted human (or automation) performing support/ops — identified by a staff DID on Substratum-operated PDS, not a customer entitlement row alone. |
| Staff DID | Operator identity | did:plc:… account hosted on pds.substratum.cloud used for company SSO; mapped to operator roles in Postgres. |
| Company SSO | Staff authentication | AT Protocol OAuth against Substratum-operated PDS — same profile as customer login (ADR 12, crates/auth) with a separate ops session and role allowlist. |
| Operator role registry | AuthZ table | Postgres mapping staff_did → role(s) (support_read, entitlement_mutator, billing_reconciler, …); source of truth for NR2. |
| Admin tooling | Operator surface | apps/admin UI on admin.* and internal operator admin API used to inspect and mutate entitlements and support state — no standalone operator CLI. |
| Manual billing adapter | Billing port impl | Admin actions that emit the same normalized events as PSP webhooks (ADR 37). |
| Entitlement audit log | Append-only record | Who changed which DID's capabilities/plan, when, why, and correlation ids. |
| Support seat | Discourse access | Paid-login customer access to self-hosted Discourse at support.* (Business model). |
| Support host | Customer forum origin | support.* (e.g. support.substratum.cloud) — where Substratum hosts the Discourse instance for paid login customers. |
Canonical product vocabulary: Glossary.
Context
ADR 32 CC10 and Deferred table name SaaS admin console but leave it unspecified. ADR 37 requires billing adapters (including manual admin) and audited admin tools (C4) to mutate entitlement rows — without defining operator workflows.
Business model promises Discourse for Substratum Once and base membership customers, with access after checkout. Before automated checkout and regional PSPs ship, operators must:
- Grant Once or base after manual payment verification.
- Fix mistaken lapse, grace extensions, and wrong plan_code assignments.
- Inspect usage (
used_bytes, capabilities) and sync failures (ADR 35) when customers report metadata or safety-net issues. - Issue or revoke Discourse seats consistently with entitlement state.
Customer-facing /account and GET /api/v1/me/limits (ADR 32) are not operator tools. This ADR defines internal support and entitlement admin boundaries.
Substratum already operates pds.substratum.cloud for customer Substratum login (ADR 37). Company SSO for operators reuses that PDS — staff get handles on the same operated PDS, sign in through AT Protocol OAuth, and are authorized only when their did appears in the operator role registry. No separate Google/Okta OIDC stack for v1.
Requirements
Functional requirements
| ID | Requirement |
|---|---|
| R1 | Substratum SHALL provide an operator admin API (internal, authenticated) to read entitlement state for a customer: did, handle(s), plan_code, capabilities (metadata_write_allowed, safety_net_allowed), used_bytes / reserved_bytes, lapse/grace, payment_provider, external customer/subscription ids (ADR 37 R15). |
| R2 | Lookup SHALL support at least did, handle (Substratum PDS), external_customer_id, and support email (when stored on entitlement or checkout record). |
| R3 | Admin write actions SHALL implement the manual billing adapter — emitting normalized events (checkout.completed, subscription.canceled, manual.grant, manual.grace_extend, …) that upsert entitlement rows through the same code path as PSP webhooks (ADR 37 R3–R4). |
| R4 | Supported operator mutations SHALL include at minimum: grant Substratum Once, grant/renew base membership, revoke safety-net (keep Once metadata), full lapse (capabilities per ADR 37 CC7), extend grace period, and quota/support override (documented reason, optional expiry). |
| R5 | Every admin write SHALL append an entitlement audit log row: operator_did (staff DID from ops session) or machine principal id, timestamp, action, before/after snapshot (or diff), reason/ticket_ref (required), optional PSP correlation id. |
| R6 | v1 (Garage): Substratum SHALL ship an operator UI (R7, R7a) and a documented ops runbook in docs/operations/ (Garage v1 rollout). Operators SHALL mutate entitlements through the admin API and admin.* UI — not a CLI. Mutations MUST go through the manual billing adapter (no ad-hoc SQL that bypasses audit). |
| R7 | v1 (Garage): An internal operator UI — isolated Mithril app apps/admin on admin.* origin (ADR 33 §8) — SHALL consume the same admin API as R1–R5. Reuse @substratum/ui-kit and Midnight Gallery tokens; operator routes, services, and Lingui catalogs MUST live under apps/admin only (no imports from apps/file-explorer/src). |
| R7a | Operator UI user-visible copy SHALL use Lingui (ADR 14): explicit IDs in apps/admin message-ids.ts, hand-maintained messages.json per locale, compile-locales before build, LanguagePicker and the same shipped locale set as apps/file-explorer and apps/landing (en, es, fr, de, pt-BR, fil). @substratum/ui-kit receives plain string props only. |
| R8 | Support seat workflow: When an operator grants Substratum Once or base membership, the system SHALL update entitlements via the manual billing adapter; customer Discourse access is primarily via AT Proto OIDC bridge login (ADR 39). Server-side email invite MAY remain as fallback; store did + discourse_user_id idempotently. |
| R9 | When entitlement lapses (PSP webhook or admin action), the support-seat workflow SHALL revoke or archive Discourse access per product policy (grace period configurable; default documented in ops) — suspend in Discourse in addition to OIDC gate denial (ADR 39 R8). |
| R10 | Admin tooling SHALL surface customer sync health: link to GET /api/v1/me/sync-failures data (or operator equivalent) for the DID — including entitlement-denied receipt jobs (ADR 37 CC6). |
| R11 | Admin tooling SHALL not mutate customer OAuth sessions, PDS repo records, or mesh blocks directly; it changes entitlements and support seats only. Catalog/PDS fixes remain export/runbook or customer-initiated delete (ADR 37 R7). |
| R12 | Operator API errors SHALL use stable codes aligned with customer-facing entitlement errors (ADR 37 R18) plus admin-specific codes for audit/validation failures. |
| R13 | BYO login customers with safety-net-only SKUs (future) SHALL be supportable via the same lookup/mutation API with metadata_write_allowed unchanged on Substratum PDS (N/A) and safety_net_allowed toggled only. |
| R14 | Checkout automation (ADR 36) when live SHALL call PSP billing adapters; admin tooling remains for exceptions, refunds, and pre-PSP Garage operations — not duplicate checkout UX. |
| R15 | Admin read API SHALL expose recent audit log entries for the DID (paginated) for support continuity across shifts. |
| R16 | OpenAPI models for operator DTOs (if exposed via ingress) SHALL live under crates/ingress/src/models in an admin or operator namespace; customer DTOs unchanged (ADR 32 NR3). |
| R17 | v1 (Garage): Operator UI and browser admin API SHALL authenticate staff via AT Protocol OAuth against Substratum-operated PDS (pds.substratum.cloud) — company SSO. Reuse crates/auth OAuth wiring with a separate OAuth client id / redirect for the admin.* origin; issue a dedicated substratum_ops_session cookie (or equivalent Bearer JWT) — MUST NOT reuse customer substratum_session. |
| R18 | Gateway SHALL maintain an operator role registry (Postgres): only staff_did rows with active role(s) authorize admin routes (NR2). Provisioning/removal of staff DIDs is an ops procedure (runbook); not self-service from customer UI. |
Non-functional requirements
| ID | Requirement |
|---|---|
| NR1 | Security: Operator API MUST NOT accept customer substratum_session cookies. Staff authenticate via Substratum PDS OAuth (R17) into substratum_ops_session; gateway verifies staff_did + role before admin handlers (R18). Automation MAY use a scoped machine credential (service token) against the admin API with its own audit principal — not shared browser SSO. |
| NR2 | Least privilege: Role separation — e.g. read-only support vs entitlement mutator vs billing reconciler. |
| NR3 | Audit immutability: Audit log rows MUST NOT be updated or deleted by application code; corrections are new audit entries. |
| NR4 | Cohesion: Manual billing adapter logic MUST live in a dedicated module/crate behind a port — not duplicated in SQL scripts (ADR 02). |
| NR5 | i18n: Operator UI (v1) MUST follow ADR 14 — no English-only shortcut. Every new operator string updates all shipped locale catalogs in the same change. Runbook copy MAY remain English. |
| NR6 | Observability: Operator actions emit structured logs linked to ticket_ref; no PAN or webhook secrets in logs. |
| NR7 | Lean team: Prefer API + runbook + small UI over a full CRM; Discourse remains the customer channel (Business model). |
Constraints
| ID | Constraint |
|---|---|
| C1 | Admin tooling MUST NOT be bundled in apps/file-explorer customer routes (ADR 33). |
| C2 | Customer substratum_session JWTs and unregistered DIDs (including paying customers on Substratum PDS) MUST NOT authorize admin endpoints (ADR 32 NR1). Only staff_did entries in the operator role registry grant access after PDS OAuth (R17–R18). |
| C3 | Discourse API tokens MUST NOT be exposed to browsers; invite/revoke runs server-side only. |
| C4 | Entitlement mutations without reason/ticket_ref MUST be rejected (HTTP 400). |
| C5 | Admin tooling MUST NOT call payment providers to mutate entitlements on the hot path; refunds/chargebacks are PSP-console ops that trigger manual adapter events after human verification (ADR 37 C7). |
| C6 | v1 SQL runbooks MUST NOT remain the long-term source of truth once the operator admin API ships — runbooks describe UI/API flows, not raw UPDATE entitlement. |
| C7 | Support tooling MUST NOT grant unlimited safety-net without explicit override type and expiry (prevents silent COGS leak). |
| C8 | Public internet exposure of operator API without PDS OAuth + role registry (and preferably VPN or allowlist) is forbidden. |
| C9 | Operator UI MUST NOT share apps/file-explorer routes or catalogs; it is apps/admin with its own lingui.config.ts, message-ids.ts, and per-locale messages.json kept in parity (ADR 14). MAY import @substratum/ui-kit only — MUST NOT import apps/file-explorer/src (ADR 33 §8). |
| C10 | v1: Staff operator identities MUST be dids hosted on Substratum-operated PDS (pds.substratum.cloud). BYO Bluesky handles and third-party OIDC (Google, Okta) MUST NOT gate admin tooling in v1. |
Cross-cutting challenges
| ID | Challenge | Mitigation |
|---|---|---|
| CC1 | Grant before DID exists (checkout then signup) | Store pending entitlement keyed by email + checkout id; attach to DID on first signup or OAuth link; admin UI shows pending rows. |
| CC2 | Handle ≠ Discourse username | OIDC sub = did (ADR 39); operator verifies identity in the forum before sensitive entitlement mutations. |
| CC3 | Operator fixes lapse but worker queue full | Admin action sets capabilities immediately; document retry sync-failures for customer; link ADR 35 account view. |
| CC4 | Two operators concurrent edit | Optimistic locking on entitlement row version or updated_at; second write fails with conflict code. |
| CC5 | Stripe + manual both grant base | Idempotent adapter events keyed by source + external_id; audit shows both lines. |
| CC6 | Scope creep into full billing UI | Admin tooling adjusts entitlements and support seats only; invoices/refunds stay in PSP dashboards (C5). |
| CC7 | Staff and customer accounts share one PDS | Role registry separates authZ; customer OAuth success ≠ admin access; audit always records operator_did. |
Decision
1. Architecture
- Operator admin API — Internal HTTP on gateway (or sibling ops service) under e.g.
/internal/v1/ops/...— not in public OpenAPI dump for customers. - Company SSO — Staff sign in via AT Protocol OAuth to
pds.substratum.cloud; gateway validatessubstratum_ops_sessionand operator role registry (R17–R18). Reusescrates/authwith ops-specific OAuth client metadata. - Manual billing adapter — Shared module with PSP adapters; admin writes = normalized events + audit.
- Entitlement audit log — New table(s) append-only; RLS bypass via admin DB role only inside adapter;
operator_didon every mutation. - Discourse adapter — Server-side admin API for lapse deactivate; optional email invite fallback (ADR 13, C3).
- OIDC bridge — Customer Discourse login via
id.support.*(ADR 39); AT Proto OAuth + entitlement gate.
2. Operator authentication (Substratum PDS as company SSO)
| Surface | v1 mechanism |
|---|---|
| Operator UI | AT Protocol OAuth → pds.substratum.cloud → ops callback → substratum_ops_session; staff_did must exist in operator role registry |
| Admin API (browser) | Same substratum_ops_session + role check on every request |
| Automation | Scoped machine credential (service token) against the admin API with distinct audit principal; optional VPN — not a substitute for human operator_did in UI |
| Role provisioning | Ops runbook: create staff handle on Substratum PDS, insert operator_role row; revoke by deleting/disabling role row |
Unregistered DIDs and customer substratum_session MUST NOT authorize admin routes (C2). Paying customer accounts on the same PDS remain customer-only unless explicitly provisioned as staff.
3. v1 delivery (Garage)
All rows below are required for v1 — not optional follow-ups. Build order is suggested for implementation sequencing only.
| Step | Deliverable |
|---|---|
| A | entitlement_audit_log schema; operator_role registry schema; manual billing adapter module; ops runbook in docs/operations/ (includes staff DID provisioning) |
| B | Ops OAuth routes + substratum_ops_session; operator admin API (lookup + write mutations + audit history + sync-failures summary) |
| C | Discourse at support.* + OIDC bridge at id.support.* (ADR 39); lapse deactivate hook |
| D | apps/admin on admin.*: isolated Vite app; PDS company SSO; Lingui + 6 locales; @substratum/ui-kit + Midnight Gallery; app-local operator pages (lookup, grant, audit) |
Step A unblocks ADR 37 Garage ops before Stripe (ADR 32 CC10). Garage v1 is not complete until step D ships.
4. Post-v1
| Item | Deliverable |
|---|---|
| E | Pending-entitlement-by-email for pre-signup checkout |
| F | Machine-credential rotation automation; optional break-glass staff DID policy |
5. Discourse integration (normative behavior)
Discourse runs on support.* (e.g. support.substratum.cloud) — a dedicated host separate from admin.* (operator UI) and app.* (customer app). Customer login uses the OIDC bridge at id.support.* (ADR 39): AT Proto OAuth upstream, entitlement gate, standard OIDC toward Discourse. The billing adapter invokes the Discourse Admin API server-side for lapse suspend; optional email invite remains a fallback (C3).
| Entitlement event | Discourse action |
|---|---|
| Grant Once or base | Entitlement updated; customer signs into Discourse via Substratum OIDC; store discourse_user_id linked to did; optional email invite fallback |
| Lapse (after grace) | Suspend user and remove from paid-member group (policy TBD in ops runbook) |
| Manual revoke support only | Operator action revoke_support_seat without changing Once/base capabilities |
Invite failures MUST NOT roll back entitlement grant; log discourse.invite_failed on audit row for operator follow-up (CC2).
6. Relationship to customer surfaces
| Surface | Audience | Purpose |
|---|---|---|
/account, /me/limits on app.* | Customer | Plan, usage, CTAs |
/account/sync-failures on app.* | Customer | Failed receipt/catalog jobs |
support.* | Customer (paid login) | Discourse — external to file-explorer; OIDC via id.support.* (ADR 39) |
app.* /account | Customer | Plan, usage, CTAs; link to support.* when entitled — no in-app Discourse |
admin.* — operator API / UI | Staff | Lookup, grant, lapse, audit, Discourse seat management |
Operators use admin.*; customers use support.* + account pages — no shared UI.
Rejected alternatives
| Alternative | Why rejected |
|---|---|
| Raw SQL as permanent mutation path | No audit, bypasses manual billing adapter (C6). |
| Admin pages inside file-explorer | Violates C1; customer bundle size and auth confusion. |
| Discourse-only ops (no entitlement API) | Cannot reconcile capabilities with ADR 37 proxy. |
| CLI/runbook-only v1 (no operator UI) | Garage v1 requires operator UI (R6, R7); mutations go through admin API + UI, not a CLI. |
| Generic OIDC (Google/Okta) for staff v1 | Violates C10; dogfood Substratum PDS company SSO instead of parallel identity plane. |
| Static bearer as primary human operator auth | No per-operator operator_did in audit; superseded by PDS OAuth + role registry (R17–R18). |
| Auto-invite Discourse before payment verified | Violates business model (invite after checkout/grant). |
| Full in-house billing console | Scope creep; PSP handles invoices (CC6, C5). |
Consequences
Positive
- Closes ADR 32 CC10 admin-console gap: v1 ships operator API, audit, Discourse hooks, PDS company SSO, and localized operator UI.
- Dogfoods Substratum-operated PDS for staff identity — same stack customers use for Substratum login.
- Garage revenue path without Stripe: operator grant → capabilities → Discourse → customer signup.
- Audit trail for trust (“who granted Once to this DID?”).
Negative
- Operator UI, staff auth, Discourse automation, and audit schema are ongoing ops burden.
- Pending-entitlement-by-email adds schema and edge cases (CC1).
Neutral
- Runbook complements the operator UI for procedures and incident response; Garage v1 launches with fully localized operator UI (R7a, NR5).
- PSP webhooks and manual adapter coexist with shared audit shape (CC5).
Verification
| Scenario | Expected |
|---|---|
| Operator grants Once with ticket ref | Capabilities updated; audit row; Discourse invite queued |
| Operator lapse without reason | HTTP 400 |
| PSP webhook + manual grant same DID | Idempotent; two audit rows, one effective state |
| Read-only role calls grant | HTTP 403 |
Customer substratum_session on admin route | HTTP 401/403 |
| Customer DID OAuth on ops UI (no role row) | OAuth completes but admin API returns HTTP 403 |
| Staff PDS OAuth on operator UI | substratum_ops_session issued; staff handle in chrome; audit rows use operator_did |
| Lapse after grace | Discourse revoke invoked; putRecord denied by proxy |
Lookup by external_customer_id | Returns entitlement + audit tail |
Operator UI LanguagePicker | Switches locale; all operator chrome updates without reload (ADR 14) |
Related
- Glossary
- Business model — Discourse, Once/base support
- ADR 02: Ports and Adapters
- ADR 12: AT Protocol Rust Ecosystem — OAuth reused for PDS company SSO
- ADR 14: Frontend Internationalization — Lingui, catalogs, ui-kit boundaries (operator UI)
- ADR 32: Account Entitlements and Hosting Policy
- ADR 33: Frontend Modular Monolith
- ADR 35: Drive Node Delete — sync-failures
- ADR 36: Marketing Landing Page
- ADR 37: PDS Entitlement Proxy and Billing Adapter
- ADR 39: AT Protocol OIDC Bridge for Discourse
- Garage v1 rollout — implementation and launch sequence