Pulp Engine — Self-Hosted Deployment Guide

Covers runtime requirements, environment variables, Docker deployment (recommended), database setup, build/start, reverse proxy, logging, backup, migration, and production risks. Pulp Engine is operator-managed: you provision the infrastructure, configure credentials, handle upgrades, and run it on infrastructure you control.

---

Deployment Topologies

Choose a topology before configuring environment variables.

Topology	STORAGE_MODE	ASSET_BINARY_STORE	Template store	Asset binary store	API instances
Single-instance, no database	file	filesystem	Filesystem (TEMPLATES_DIR)	Filesystem (ASSETS_DIR)	1 only
Single-instance, database	postgres or sqlserver	filesystem	Database	Filesystem (ASSETS_DIR)	1 recommended unless ASSETS_DIR is shared
Multi-instance, database + NFS	postgres or sqlserver	filesystem	Database	Shared network volume (ASSETS_DIR)	2+
Multi-instance, database + S3	postgres or sqlserver	s3	Database	S3-compatible bucket	2+ (no shared volume required)

STORAGE_MODE controls where template and asset metadata is stored. ASSET_BINARY_STORE controls where binary files (uploaded images) are stored. They are independent — any combination of storage mode and binary store is valid.

• ASSET_BINARY_STORE=filesystem (default): Binary files are written to ASSETS_DIR on the local filesystem. In single-instance deployments this is simple and requires no extra infrastructure. In multi-instance deployments all API instances must mount the same ASSETS_DIR via a shared network volume (e.g. NFS). Without a shared volume, instance A cannot serve assets uploaded by instance B, causing 404 errors in rendered PDFs. See § 9 Known Production Risks.

• ASSET_BINARY_STORE=s3: Binary files are written to an S3-compatible bucket. Eliminates the shared-volume requirement for multi-instance deployments. See § Object Storage for configuration.

File mode in production is appropriate for a single containerised instance. It is not suitable for horizontal scaling. See § 9 Known Production Risks for the file-mode risk entry.

Quick-reference compose matrix

Topology	Storage	Asset Binary	Render	Compose File
Single instance, no DB	file	filesystem	child-process	compose.yaml
Single instance, Postgres	postgres	filesystem	child-process	compose.postgres.yaml
Multi-instance, Postgres + S3	postgres	s3	child-process	(custom — see sections below)
Privilege-separated render	file or postgres	filesystem or s3	socket	compose.container.yaml
Kubernetes, database + shared assets	postgres or sqlserver	s3 or RWX filesystem	child-process	Helm chart

The compose files are evaluator-ready starting points. For hardened production configuration, continue with the sections below.

---

1. Runtime Requirements

Requirement	Version	Notes
Node.js	22–24	node --version to confirm
pnpm	10.32.1	pnpm --version to confirm
PostgreSQL	14+ (16 recommended)	Required only when STORAGE_MODE=postgres (default); must be reachable from the API process
SQL Server	2019+ / Azure SQL	Required only when STORAGE_MODE=sqlserver; must be reachable from the API process
Chromium / Puppeteer	Bundled	Puppeteer downloads Chromium on pnpm install; ensure network access or pre-cache

Linux note: Puppeteer requires shared libraries that are not always present on minimal server images. Install the following if Chromium fails to launch:

# Debian / Ubuntu
apt-get install -y libatk1.0-0 libatk-bridge2.0-0 libcups2 \
  libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 \
  libxrandr2 libgbm1 libasound2

Chrome sandbox: The Chrome sandbox is enabled by default. Only --disable-dev-shm-usage and --disable-gpu are always passed to Puppeteer. The --no-sandbox and --disable-setuid-sandbox flags are added only when PULP_ENGINE_DISABLE_SANDBOX=true is set in the environment. Default container deployments set this variable; Kubernetes can opt into a node-supported user-namespace sandbox as described in the Kubernetes guide. Containers without that setup must set it — Chrome will fail to launch in a container without it.

---

2. Environment Variables

Copy .env.example to .env and set each variable before starting the process.

Running tests locally? See ../apps/api/README.md for the dev-loop setup (repo-root .env, throwaway Postgres, db:deploy, run the suite).

Table columns. Default is what the runtime uses when the variable is absent (or explicitly unset). Example value shows a non-default setting. Required calls out the modes in which omission causes startup to fail.

Variable	Required	Default	Example value	Description
DATA_SOURCE_MAX_RESPONSE_BYTES	No	10485760	2097152	Maximum response bytes per template source, summed across pagination pages; configurable from 1 KiB to 100 MiB.
DATA_SOURCE_ALLOWED_ORIGINS	No	empty	https://crm.internal.example	Comma-separated exact trusted HTTP(S) origins that may reach private addresses. DNS is still pinned; production requires HTTPS. Connection paths remain contained by their base URL.
STORAGE_MODE	No	postgres	postgres	Storage backend: postgres (default), sqlserver, or file. Plugin-provided backends are also accepted — the plugin system validates at activation time. Selects which adapter is loaded at startup.
DATABASE_URL	Yes (postgres mode)	—	postgresql://user:pass@host:5432/pulp-engine?schema=public	Prisma connection string. Required when STORAGE_MODE=postgres; not read otherwise.
SQL_SERVER_URL	Yes (sqlserver mode)	—	mssql://user:pass@host:1433/pulp-engine?encrypt=false&trustServerCertificate=true	mssql connection URL. Required when STORAGE_MODE=sqlserver; not read otherwise.
BATCH_ASYNC_DURABILITY	No	required (hardened) / warn (non-hardened)	warn	Tri-state gate — required / warn / allow-nondurable. When STORAGE_MODE=file, controls whether the API refuses boot, logs a warning, or boots quietly given that async-batch state is in-memory only in file mode. Default mirrors REQUIRE_HTTPS: hardened production sets the default to required (refuses boot) and operators must explicitly downgrade. See § Operational Limitations vs Postgres.
TEMPLATES_DIR	Yes (file mode)	—	./templates	Path to a directory of TemplateDefinition JSON files. Required when STORAGE_MODE=file; not read in database modes.
HOST	No	0.0.0.0	0.0.0.0	Bind address. Defaults to 0.0.0.0. Use 127.0.0.1 if behind a reverse proxy on the same host.
PORT	No	3000	3000	Listener port. Defaults to 3000.
NODE_ENV	No	development (Docker image: production)	production	Accepted values: development, production, test. Application default is development; the Docker image sets NODE_ENV=production in its final stage. In production mode, JSON log output is enabled and hardening enforcement is active by default (see HARDEN_PRODUCTION).
API_KEY_ADMIN	Yes (production, recommended)	—	a-long-random-string	Admin-scoped credential — full access: templates, assets, render, preview. At least one of API_KEY_ADMIN, API_KEY_RENDER, or the deprecated API_KEY must be set in production — unless using named-user-only mode (EDITOR_USERS_JSON without any shared API keys), which is a valid standalone configuration.
API_KEY_RENDER	No	unset	another-long-random-string	Render-scoped credential — all production render routes: POST /render/pdf (canonical; POST /render is a deprecated alias), /render/html, /render/csv, /render/xlsx, /render/docx, /render/pptx, /render/batch, /render/batch/docx, /render/batch/pptx, and /render/pdf/*. Use for production integrations.
API_KEY_PREVIEW	No	unset	yet-another-random-string	Preview-scoped credential — POST /render/preview/* only.
API_KEY_EDITOR	No	unset	a-different-random-string	Editor-scoped credential — template management, asset management, POST /render/preview/*, and POST /render/validate. Operators enter this value in the editor’s interactive login form (v0.15.0 — no VITE_API_KEY or frontend env var required). Cannot call any production render route or admin-only operations (delete template, restore version, promote label, admin routes).
API_KEY	Deprecated	—	—	Legacy single key (treated as admin scope). Accepted for migration; cannot be set alongside the new scoped keys. See § Migration below.
EDITOR_TOKEN_TTL_MINUTES	No	480	480	Lifetime of a minted editor session token in minutes. Default: 480 (8 hours). Accepted range: 5–1440. Shorter values reduce the exposure window if a token is compromised. Requires an API restart to take effect.
EDITOR_TOKEN_ISSUED_AFTER	No	unset	2026-03-24T12:00:00Z	Reject any editor session token whose issued-at timestamp is before this UTC datetime. Accepts a UTC ISO-8601 string. Use to invalidate all outstanding editor sessions without rotating API_KEY_EDITOR. Requires an API restart to take effect. In multi-instance deployments server clocks must be reasonably synchronised (NTP). Pre-v0.19.0 tokens are always rejected when this is set.
API_KEY_EDITOR_PREVIOUS	No	unset	—	Verify-only previous editor key for near-zero-downtime session rotation. Set to the old API_KEY_EDITOR value while setting the new value as the active key. Outstanding editor session tokens signed with the old key continue to verify; only the new key mints. Does not preserve direct X-Api-Key usage of the old key — callers using the old key directly must switch to the new key during the rollout. Cannot be used as X-Api-Key or to mint tokens. Must not equal any active key. Remove after EDITOR_TOKEN_TTL_MINUTES elapses and restart again. See runbook.md § Auth secret rotation. Requires an API restart.
API_KEY_ADMIN_PREVIOUS	No	unset	—	Verify-only previous admin key. Same semantics as API_KEY_EDITOR_PREVIOUS but for editor session tokens signed with API_KEY_ADMIN. Does not preserve direct X-Api-Key usage of the old admin key. See runbook.md § Auth secret rotation. Requires an API restart.
EDITOR_USERS_JSON	No	unset (shared-key mode)	[{"id":"alice",...}]	Per-user named credential registry (v0.23.0+). When set, the editor login gate operates in named-user mode: each operator has a personal key, server-derived identity, and an optional role. See § Named-User Mode for the full format and migration runbook. When absent, shared-key mode is active (existing behavior). Requires an API restart when changed.
ASSET_BINARY_STORE	No	filesystem	filesystem	Where uploaded asset binaries (images) are stored. filesystem (default) writes to ASSETS_DIR; s3 stores in an S3-compatible bucket. Independent of STORAGE_MODE. See § Object Storage.
ASSET_ACCESS_MODE	No	public	public	Controls how asset binaries are delivered. public (default) serves assets without auth — static file serving (filesystem) or public S3 URLs. private routes all asset delivery through an authenticated API proxy. See § Asset Access Mode.
ASSETS_DIR	No	./assets	./assets	Filesystem path where uploaded image assets are stored. Used only when ASSET_BINARY_STORE=filesystem. The directory is created automatically on startup if it does not exist. Use an absolute path in production (e.g. /var/pulp-engine/assets).
ASSETS_BASE_URL	No	/assets	/assets	URL prefix under which asset files are served (e.g. /assets/uuid-logo.png). Used only when ASSET_BINARY_STORE=filesystem. Must match the path configured in your reverse proxy if assets are proxied separately.
ASSET_SVG_UPLOADS_ENABLED	No	true	false	Accept new SVG uploads after sanitization. Disable to reject SVG uploads and replacements; legacy unsanitized SVG stays quarantined.
ASSET_SVG_MAX_BYTES	No	2097152	2097152	SVG input byte limit (1 to 10485760), in addition to XML depth/element caps.
ASSET_VERSION_RETENTION	No	10	10	Previous binary versions retained per asset (0 to 100); replacement and restore keep the same URL.
ASSET_DELETE_GUARD	No	true	false	Block deletion when current or label-pinned templates reference an asset. Explicit force is audited. Disable for older unguarded semantics.

Asset upload validation: PNG, JPEG, GIF and WebP use MIME/magic-byte validation. SVG uses bounded XML parsing and a closed allow-list that strips executable content and external references; malformed XML and DTD/entities return 415. Only newly sanitized SVG is served. Legacy unsanitized SVG stays blocked at API proxy/static/inline/validation boundaries; use GET /assets?legacySvg=true for remediation. Public S3 bypasses API CSP protection: use private delivery or equivalent bucket/CDN response headers, and deny public access to .versions/ history keys.

Variable	Required	Default	Example value	Description
S3_BUCKET	Yes (s3 mode)	—	my-pulp-engine-assets	S3 bucket name. Required when ASSET_BINARY_STORE=s3.
S3_REGION	Yes (s3 mode)	—	us-east-1	AWS region (or an arbitrary string for MinIO — MinIO requires a non-empty value). Required when ASSET_BINARY_STORE=s3.
S3_ACCESS_KEY_ID	Yes (s3 mode)	—	AKIA...	AWS access key ID. Required when ASSET_BINARY_STORE=s3.
S3_SECRET_ACCESS_KEY	Yes (s3 mode)	—	—	AWS secret access key. Required when ASSET_BINARY_STORE=s3.
S3_ENDPOINT	No	unset	https://minio.example.com	S3-compatible API endpoint for MinIO, Cloudflare R2, or other custom providers. Omit for standard AWS S3. When set, S3_PUBLIC_URL is required (the API endpoint and public delivery URL differ for custom providers).
S3_PATH_STYLE	No	false	true	Set to true to force path-style URLs (required for MinIO). Default: false. When true, S3_PUBLIC_URL is required.
S3_PUBLIC_URL	Yes (custom endpoint or path-style, public mode only)	auto-derived for AWS public mode	https://assets.example.com	Base URL for public asset delivery. Required when S3_ENDPOINT is set or S3_PATH_STYLE=true and ASSET_ACCESS_MODE=public. Not required when ASSET_ACCESS_MODE=private. Optional for standard AWS S3 in public mode — auto-derived as https://{bucket}.s3.{region}.amazonaws.com. Trailing slash is stripped automatically.
PULP_ENGINE_DISABLE_SANDBOX	No	unset (sandbox enabled)	true	Set to true in container environments (Docker, Kubernetes) where the Chrome sandbox is unavailable. Default: unset — sandbox enabled. See § 1 Runtime Requirements.
PREVIEW_ROUTES_ENABLED	No	false (absent → 404 in production)	true	Registers the editor preview routes (POST /render/preview/html, POST /render/preview/pdf) in production. Default: absent — routes return 404. Set to true only when the visual editor must reach the API directly in production for live visual previews. Ignored when NODE_ENV is not production. Pair with network-level restrictions when enabled. Not required for the publish flow — POST /render/validate (publish-readiness validation) is always registered, never returns rendered output, and is safe to keep enabled in production even when preview routes are disabled.
LOG_LEVEL	No	info	info	Pino log level. One of trace, debug, info, warn, error. Default: info. Use warn in high-throughput environments to reduce log volume.
CORS_ALLOWED_ORIGINS	No	allow all (with production warning)	https://editor.example.com	Comma-separated list of origins allowed to make cross-origin browser requests to this API. Origins must include the scheme and exact hostname (e.g. https://editor.example.com). Use * to allow all origins explicitly. When absent, all origins are allowed (origin: true) with a production startup warning. Same-origin deployments (editor and API on the same host) do not require this setting.
DOCS_ENABLED	No	true (Docker image: false)	false	Set to false to skip registering Swagger UI (/docs, /docs/json, /docs/yaml). All /docs* routes return 404. Application default: true (backward-compatible). Docker image default: false — the container image ships with docs disabled; set DOCS_ENABLED=true to re-enable. Disable in production to reduce the exposed API surface if the interactive docs are not needed by operators.
METRICS_TOKEN	No	unset (unauthenticated)	<random-hex>	When set, GET /metrics requires an Authorization: Bearer <token> header. When absent, the endpoint is unauthenticated (current behavior, preserved for backward compatibility). Generate with openssl rand -hex 32. Operators should restrict at the network layer regardless of whether a token is set.
TRUST_PROXY	No	false	true	Set to true when the API is behind a reverse proxy that sets X-Forwarded-Proto. Enables Fastify trustProxy so request.protocol accurately reflects the connection scheme. Required for REQUIRE_HTTPS to work correctly. Default: false.
REQUIRE_HTTPS	No	false	true	When true, POST /auth/editor-token rejects non-HTTPS requests with 400 Bad Request, preventing credentials from being sent over plain HTTP. Requires TRUST_PROXY=true and a TLS-terminating reverse proxy. Default: false.
HARDEN_PRODUCTION	No (auto-derived)	auto-derived from NODE_ENV	false	Controls fail-fast enforcement of security controls. Default: auto-derived from NODE_ENV — enforced when NODE_ENV=production, off otherwise. Set HARDEN_PRODUCTION=false to explicitly opt out for evaluation. Accepted values: true, false, 1, 0, or unset. See Hardened Production Mode.
BLOCK_REMOTE_RESOURCES	Yes (hardened mode)	false	true	When true, the render pipeline blocks all outbound http/https fetches during PDF generation except origins listed in ALLOWED_REMOTE_ORIGINS. Required in hardened production mode (v0.54.0+).
ALLOWED_REMOTE_ORIGINS	No	unset (no origins allowed)	https://fonts.googleapis.com,https://fonts.gstatic.com	Comma-separated URL origins permitted when BLOCK_REMOTE_RESOURCES=true. Each entry must be a valid URL origin (scheme + host + optional port). Has no effect when BLOCK_REMOTE_RESOURCES is not true.
ALLOW_SHARED_KEY_EDITOR	No	false	true	When true, explicitly accepts shared-key identity for editor login in hardened production, bypassing the EDITOR_USERS_JSON requirement. Use when configuring a user registry is not practical. Default: false.
ALLOW_NO_AUTH	No (dev/test only)	false	true	Local development / testing only. When authentication resolves to fully disabled (no API_KEY_* and no named-user registry), the server refuses to boot unless this is true. NODE_ENV-independent — closes the footgun where a bare node dist that forgets NODE_ENV=production would run unauthenticated. Never set in production; configure a credential (API_KEY_*) or a named-user registry (EDITOR_USERS_JSON/EDITOR_USERS_FILE/EDITOR_USERS_DB) instead. Accepted values: true, false, 1, 0, or unset.
RENDER_MODE	No	child-process	child-process	Controls where Puppeteer executes during PDF rendering. child-process (default): persistent child process with empty env — no secrets reachable. container: ephemeral Docker container per render — stronger isolation, API process holds Docker socket. socket: render requests routed through a separate controller process — API holds no Docker socket authority. in-process: Puppeteer runs inside the API process — legacy, debugging only. See § Render Isolation Mode.
RENDER_CONTAINER_IMAGE	Yes (container mode)	—	ghcr.io/OWNER/pulp-engine-worker:vX.Y.Z	OCI image for the worker container. Required when RENDER_MODE=container. Build with docker build -f Dockerfile.worker -t <image> ..
PULP_ENGINE_PLUGINS	No	unset	pulp-engine-plugin-barcode	Comma-separated list of plugin package names or file paths. Plugins are loaded after core initialization and can extend renderers, storage backends, auth providers, routes, and events. Enabled plugins run as trusted server code inside the API process — see plugin-trust-model.md before enabling third-party plugins. See examples/plugin-barcode for a reference implementation.
RENDER_CONTROLLER_SOCKET	Yes (socket mode)	—	/run/render/render.sock	Path to the Unix domain socket created by the render-controller process. Required when RENDER_MODE=socket. See § Render Isolation Mode and compose.container.yaml.
MCP_HTTP_ENABLED	false	Expose the stateless POST /mcp agent endpoint. API-key/editor-token callers retain their existing scopes.
MCP_HTTP_TOOLS	read	read, write or admin: maximum registered tool set, still checked against each caller’s inner API scope.
RATE_LIMIT_MCP_MAX	60	MCP requests per minute; follows rate-limit key mode and preserves original caller IP for inner calls.
RATE_LIMIT_MAX	No	100	100	Global rate limit (requests per minute per IP). Default: 100. Applies to all routes except health, metrics, and those with per-route overrides.
RATE_LIMIT_RENDER_MAX	No	20	20	Rate limit for render and preview endpoints (requests per minute per IP). Default: 20. Applies to POST /render/pdf (and the deprecated /render alias), /render/html, /render/csv, /render/xlsx, /render/docx, /render/pptx, /render/validate, /render/pdf/*, and /render/preview/*. Batch routes use RATE_LIMIT_BATCH_MAX.
RATE_LIMIT_BATCH_MAX	No	5	5	Rate limit for POST /render/batch (requests per minute per IP). Default: 5. Separate from RATE_LIMIT_RENDER_MAX because a single batch request can trigger many renders.
BATCH_MAX_ITEMS	No	50	50	Maximum number of items per POST /render/batch request. Range: 1–200. Requests exceeding this limit receive 400 Bad Request.
BATCH_CONCURRENCY	No	5	5	Maximum concurrent renders within a single batch request. Range: 1–20. Should not exceed the Chrome page pool size (MAX_CONCURRENT_PAGES = 5) to avoid starving concurrent non-batch renders.
RATE_LIMIT_STORE	No	memory	memory	Rate-limit counter backend. memory (default): in-process LRU — sufficient for single-instance deployments. redis: shared Redis counters — required for consistent per-IP enforcement across multiple API instances (in-process counters are per-replica, so the effective per-IP limit multiplies with the instance count). Requires REDIS_URL. A production startup warning is logged when memory is used with a database-backed (multi-instance-capable) storage mode. The bundled docker-compose.ha.yml sets this to redis.
REDIS_URL	Yes (redis mode)	—	—	Redis connection URL for cluster-aware rate limiting. Format: redis://[:password@]host:port[/db] or rediss:// for TLS. Required when RATE_LIMIT_STORE=redis.
RATE_LIMIT_FAIL_OPEN	No	false	false	Controls behavior when RATE_LIMIT_STORE=redis and Redis becomes unreachable at runtime. false (default): requests receive 500 errors — operator notices immediately. true: rate limiting degrades to unlimited with a warning. Does not affect startup (always fail-fast).
PREVIEW_BODY_LIMIT	No	524288	524288	Maximum request body size in bytes for preview endpoints (POST /render/preview/html, POST /render/preview/pdf). Default: 524288 (512 KiB). Increase if templates with large inline definitions exceed the limit.
APP_VERSION	No	package.json version	0.68.0	Overrides the version field reported by GET /health and GET /health/ready. Auto-set by the Docker build (ARG); defaults to package.json version when unset. Operators only need to set this for custom non-Docker builds.

Opt-in capability flags

These environment variables enable features that are off by default. Each is independently gated.

Variable	Required	Default	Example value	Description
MULTI_TENANT_ENABLED	No	false	true	Turns on per-tenant isolation of templates, assets, credentials, audit events, and schedules. Requires a database-backed storage mode (Postgres or SQL Server); rejected at startup for file mode. See § Multi-Tenant Mode.
API_KEYS_JSON	Yes (multi-tenant)	—	[{"key":"...","scope":"admin","tenantId":"acme"}]	Tenant-scoped credential registry used when MULTI_TENANT_ENABLED=true. Replaces the flat API_KEY_* variables in multi-tenant mode. A super-admin entry (tenantId: null) is required for /admin/tenants CRUD. Single-tenant deployments may still use this variable, but every entry must be tenantId: "default".
API_KEY_SUPER_ADMIN	No	unset (rejected in single-tenant mode)	<random-hex>	Legacy single super-admin credential. Only valid when MULTI_TENANT_ENABLED=true; rejected at startup in single-tenant mode since v0.67.1. Prefer an API_KEYS_JSON entry with tenantId: null.
TENANT_STATUS_CACHE_TTL_MS	No	10000	10000	Cache TTL for tenant archive/active state. Default 10 000 ms. Lower values reduce the window where an archived tenant can still be written to on non-handling pods.
SCHEDULE_ENABLED	No	false (engine off — schedules never fire)	true	Starts the cron/dispatcher execution engine. Schedule CRUD routes (/schedules/*) are available whenever the storage mode provides a schedule store (Postgres or SQL Server) — with SCHEDULE_ENABLED=false schedules can be created and edited but never fire (no 503; the create succeeds and the engine simply isn’t running). File mode has no schedule store, so its routes return 503 unavailable. Requires Postgres or SQL Server (SCHEDULE_ENABLED=true is rejected at startup for file mode).
OIDC_DISCOVERY_URL	Yes (OIDC)	—	https://issuer.example.com/.well-known/openid-configuration	OIDC discovery document URL. Presence of this variable enables OIDC — there is no separate OIDC_ENABLED flag. Requires OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, and OIDC_COOKIE_SECRET. See oidc-guide.md.
OIDC_CLIENT_ID	Yes (OIDC)	—	—	OAuth 2.0 client ID registered with the identity provider.
OIDC_CLIENT_SECRET	Yes (OIDC)	—	—	OAuth 2.0 client secret.
OIDC_COOKIE_SECRET	Yes (OIDC)	—	<openssl rand -hex 32>	Secret (≥ 32 chars) used to sign OIDC session cookies.
OIDC_REDIRECT_URI	Yes (OIDC)	—	https://api.example.com/auth/oidc/callback	OAuth redirect URI. Required whenever OIDC_DISCOVERY_URL is set (startup fails otherwise); must match the value registered with the provider. In hardened mode the URI must use https://.
OIDC_SCOPES	No	openid profile email	openid profile email	Space-separated OIDC scopes requested.
OIDC_CLAIM_SUB / OIDC_CLAIM_EMAIL / OIDC_CLAIM_DISPLAY_NAME / OIDC_CLAIM_GROUPS	No	sub / email / name / groups	sub / email / name / groups	Override the ID-token claim names read during provisioning.
OIDC_ADMIN_GROUPS	No	unset (no admin promotion)	pulp-admin,pulp-owners	Comma-separated groups whose members are promoted to admin role. When unset, no user is promoted to admin.
OIDC_EDITOR_GROUPS	No (must be set explicitly in hardened mode)	* (any authenticated user)	pulp-editors	Groups whose members receive editor scope (template create/update/delete + asset write). * (default) grants editor to every authenticated SSO user; an empty string grants editor only via OIDC_ADMIN_GROUPS; a comma-separated list restricts to those groups. In hardened mode the server refuses to start if OIDC is enabled and this is left unset — choose a value deliberately. Outside hardened mode, a startup warning is logged when the default * is in effect.
OIDC_AUTO_PROVISION	No	true	true	When true, unknown OIDC users are added to the registry on first login. Persistence depends on storage mode: postgres/sqlserver persist automatically to the DB-backed editor_users registry (EDITOR_USERS_DB=true); file mode requires EDITOR_USERS_FILE (or an EDITOR_USERS_JSON seed). Without persistence, auto-provisioned users are in-memory only and a startup warning is emitted.
OIDC_DEFAULT_TENANT	No	default	default	Tenant assigned to auto-provisioned OIDC users in multi-tenant mode.
OIDC_PROVIDER_NAME	No	SSO	Okta	Display name shown on the editor login button. Default SSO.
ANTHROPIC_API_KEY	No	unset (/templates/generate not registered)	sk-ant-...	Enables POST /templates/generate (AI template generation). When unset, the route is not registered. Also reflected in GET /capabilities.
ANTHROPIC_MODEL	No	claude-opus-4-7	claude-opus-4-7	Claude model used for AI generation.
ANTHROPIC_MAX_TOKENS	No	4096	4096	Max output tokens per generation request.
EMBED_ALLOWED_ORIGINS	Yes (embed)	—	https://app.acme.com https://staging.acme.com	Space-separated origins permitted to embed the editor. Sets frame-ancestors on /embed.html responses. Without this, the embeddable editor cannot load inside a customer iframe.
EMBED_CONNECT_ORIGINS	No	unset	https://forms-proxy.acme.com	Space-separated extra origins the embedded editor may open fetch/WebSocket connections to (connect-src in the embed CSP).
EDITOR_USERS_FILE	No	unset (in-memory mutations only)	/etc/pulp-engine/users.json	Path to a named-user registry file (file mode). Enables runtime POST /admin/users/PUT/DELETE/reload to mutate users across restarts. Without this, EDITOR_USERS_JSON is read once at startup and runtime mutations are in-memory only. In postgres/sqlserver mode the registry is DB-backed instead; EDITOR_USERS_JSON/FILE then only seed an empty editor_users table on first boot (the DB is authoritative thereafter), and POST /admin/users/reload reloads the cache from the DB.
EDITOR_USERS_DB	No	false	true	(v0.81.0, postgres/sqlserver only) Enables DB-backed named-user mode with no EDITOR_USERS_JSON/FILE seed (empty-table bootstrap — create the first user with an admin API key via POST /admin/users). Counts as named-user intent for identity mode and the hardening named-user-registry control. Rejected at startup under file/plugin storage.
EDITOR_USERS_CACHE_TTL_MS	No	10000	10000	(v0.81.0) How long each instance may serve the in-memory editor-user cache before a full reload from the store. New users appear immediately (read-through on miss); role changes / tokenIssuedAfter revocations / deletes propagate to other replicas within this window. Range 1000–300000.
PULP_ENGINE_PLUGINS	No	unset	pulp-engine-plugin-barcode	Comma-separated plugin package names or file paths. Plugins can extend renderers, storage backends, auth providers, routes, and events. Trust posture: plugins run as trusted server code inside the API process — see plugin-trust-model.md. See examples/plugin-barcode.
PPTX_ENABLED	No	true	true	Enables the /render/pptx and /render/batch/pptx routes. Default: true. Set to false to withhold PPTX from /capabilities and return 404 on the routes.
USAGE_QUERY_MAX_WINDOW_DAYS	No	92	92	Maximum [from, to) window accepted by GET /usage. Default 92.
USAGE_EXPORT_MAX_ROWS	No	500000	500000	Row-count cap for GET /usage.csv. Requests exceeding this return 413 usage_export_too_large. Default 500 000.
RENDER_USAGE_RETENTION_DAYS	No	unset (retain indefinitely)	365	How long to retain per-render usage rows. Unset = retain indefinitely.
RENDER_USAGE_PURGE_INTERVAL_HOURS	No	24	24	How often the usage-retention purge runs. Default 24 hours.
PULP_LICENCE_KEY	No	unset (evaluation watermark)	<5-part signed token from your licence email>	Commercial licence key (Ed25519-signed token, verified offline). Unset or invalid → the server runs under the Evaluation Licence and rendered output carries a small watermark; rendering never refuses on a licence problem. Licence status is surfaced in /health/ready. See licence-key-format.md.
SCHEDULE_POLL_INTERVAL_SECONDS	No	15	15	How often the schedule engine polls for due schedules. Range 5–300.
SCHEDULE_MAX_CONCURRENT	No	2	2	Max schedule executions rendered concurrently per instance. Range 1–10.
ASSET_PDF_MAX_PAGES	No	50	50	Page limit for uploaded PDF stationery/insert assets; range 1-500.
BUNDLE_MAX_BYTES	No	262144000	262144000	Maximum imported template bundle bytes; range 1024-1073741824.
LABEL_CONTRACT_GATE	No	off	enforce	Require compatible contracts when moving labels or importing bundles; off or enforce.
SCHEDULE_BURST_MAX_ITEMS	No	1000	1000	Maximum selected records in one schedule burst; range 1-50000. Budget temp disk for items times average output size.
SCHEDULE_BURST_MAX_CONCURRENCY	No	4	4	Maximum concurrent records per burst; range 1-32. Schedule default is 2.
SCHEDULE_SMTP_MAX_CONNECTIONS	No	4	4	Connections in the pooled schedule SMTP transport; range 1-32.
SCHEDULE_DELIVERY_MAX_RETRIES	No	4	4	Max delivery attempts per target (email/S3/webhook). Range 1–10.
SCHEDULE_CATCHUP_WINDOW_MINUTES	No	15	15	Missed fires older than this are skipped instead of fired late (e.g. after downtime). Range 1–1440.
SCHEDULE_STALL_TIMEOUT_MS	No	300000	300000	Claimed executions older than this are reclaimed (instance died mid-render). Also bounds the startup sweep that removes stale scheduled-render temp artifacts. Range 30 000–600 000.
SCHEDULE_EXECUTION_RETENTION_DAYS	No	30	30	Completed/failed execution rows older than this are purged. Range 1–3650.
SCHEDULE_ENGINE_TICK_LIMIT	No	500	500	Max due schedules processed per engine tick (unbounded-query guard). Range 10–10 000.
SCHEDULE_SMTP_HOST	Yes (email delivery)	unset (email targets fail)	smtp.sendgrid.net	SMTP relay host for schedule email delivery targets. Without it, schedules with email targets fail delivery. nodemailer is lazy-loaded — no overhead when unset.
SCHEDULE_SMTP_PORT	No	587	587	SMTP port.
SCHEDULE_SMTP_USER / SCHEDULE_SMTP_PASS	No	unset (unauthenticated relay)	—	SMTP credentials.
SCHEDULE_SMTP_FROM	No	unset	reports@example.com	From address on delivered emails.
SCHEDULE_SMTP_FROM_NAME	No	unset	Reporting	Default display name for schedule email.
SCHEDULE_EMAIL_MAX_ATTACHMENT_BYTES	No	10000000	10000000	Auto delivery switches larger single outputs to stored links.
SCHEDULE_SMTP_SECURE	No	false (STARTTLS)	true	true = implicit TLS (typically port 465); false = STARTTLS upgrade on a plaintext connection (typically 587).
SCHEDULE_SMTP_REQUIRE_TLS	No	true	true	With SCHEDULE_SMTP_SECURE=false, require the relay to actually upgrade to TLS — a relay without STARTTLS hard-fails instead of silently transmitting credentials and the rendered attachment in cleartext. Security opt-out: an unrecognised value fails config validation.
SCHEDULE_S3_BUCKET	Yes (S3 delivery)	unset	acme-reports	Bucket for schedule S3 delivery targets. Deliberately separate from the asset-binary S3_* settings — delivery may target a different bucket/role.
SCHEDULE_S3_REGION / SCHEDULE_S3_ACCESS_KEY_ID / SCHEDULE_S3_SECRET_ACCESS_KEY / SCHEDULE_S3_ENDPOINT	No	unset	—	Credentials/region/endpoint for schedule S3 delivery (endpoint for MinIO/R2-style providers).
SCHEDULE_DLQ_ENABLED	No	true	true	When true, exhausted schedule deliveries write a reference row to schedule_delivery_dlq for /admin/schedule-dlq list/replay. false = log-only exhaustion reporting. Postgres only; the store is null in file/sqlserver modes regardless.
SCHEDULE_DLQ_RETENTION_DAYS	No	30	30	DLQ rows older than this are purged by the schedule engine’s retention tick. Range 1–365.
BATCH_WEBHOOK_DLQ_ENABLED	No	true	true	When true, exhausted async-batch webhook deliveries write a reference row to batch_webhook_dlq for /admin/batch-dlq list/replay. false = log-only. Postgres only.
WEBHOOK_MAX_CONCURRENT_JOBS	No	10	10	Max async-batch jobs processing simultaneously; submissions beyond the cap are shed with 503 + Retry-After. Range 1–50.
WEBHOOK_TIMEOUT_MS	No	10000	10000	HTTP timeout for each outbound webhook POST. Range 1 000–60 000.
WEBHOOK_MAX_RETRIES	No	3	3	Max webhook delivery attempts (including the first). Range 1–10.
JOB_POLL_HINT_SECONDS	No	2	2	Retry-After hint for accepted and active render jobs; 1-60 seconds.
JOB_INLINE_CONTENT_MAX_BYTES	No	10485760	10485760	Maximum raw bytes returned as inline base64 via ?include=content.
IDEMPOTENCY_ENABLED	No	true	true	Enable optional request idempotency on opted-in mutation routes.
IDEMPOTENCY_STORE	No	auto	database	auto, redis, database or memory; explicit Redis requires REDIS_URL, database requires PostgreSQL/SQL Server.
IDEMPOTENCY_TTL_HOURS	No	24	24	Completed response/marker retention, 1-168 hours.
IDEMPOTENCY_LOCK_TTL_SECONDS	No	120	120	In-flight lease duration, 1-3600 seconds; size above the longest accepted request.
IDEMPOTENCY_MAX_BODY_BYTES	No	262144	262144	Replayable serialized JSON response limit; larger successes retain a bodyless marker.
WEBHOOK_JOB_RETENTION_SECONDS	No	3600	3600	How long completed/failed/cancelled render jobs stay pollable (and, on PostgreSQL/SQL Server, how long their durable rows + result blobs are retained). The sweep runs every 60 s, so effective retention can exceed this by up to ~60 s. Range 60–86 400.
WEBHOOK_JOB_TIMEOUT_MS	No	300000	300000	Max wall-clock time an async-batch job may run before the sweep fails it and releases its concurrency slot. Range 30 000–600 000.
JOB_RESULT_BLOB_STORE	No	filesystem	s3	Where private job metadata envelopes (gzipped JSON) and separate item bytes are persisted on PostgreSQL/SQL Server deployments. Item keys are {tenantId}/{jobId}/items/{index}. filesystem → {JOB_RESULT_BLOB_DIR}/{tenantId}/{jobId}.json.gz — multi-instance/HA deployments require a shared volume, or use s3. s3 → s3://{JOB_RESULT_BLOB_BUCKET}/..., reusing the asset-binary S3_REGION/S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY/S3_ENDPOINT/S3_PATH_STYLE settings. The S3 credentials additionally need ListBucket for the v0.84.0 blob orphan-reconciliation sweep.
JOB_RESULT_BLOB_DIR	No	./job-result-blobs	/data/job-result-blobs	Filesystem root for result blobs (JOB_RESULT_BLOB_STORE=filesystem).
JOB_RESULT_BLOB_BUCKET	Yes (s3 blob store)	unset	acme-pulp-job-results	Bucket for result blobs (JOB_RESULT_BLOB_STORE=s3). Deliberately distinct from S3_BUCKET — job-result blobs are private, never public-ACL.
STARTUP_ORPHAN_GRACE_MS	No	60000	60000	At startup, pending/processing async-batch rows older than this are failed with job_abandoned_at_startup; younger rows are left for the just-started worker. Range 10 000–3 600 000.
RATE_LIMITS_JSON	No	unset	[{"tenantId":"acme","routeClass":"render","max":120}]	Per-tenant / per-route-class / per-template rate-limit overrides (C.2). JSON array of { tenantId, routeClass, templateKey?, max, timeWindow? }; parsed and validated at startup. Mutually exclusive with RATE_LIMITS_JSON_FILE.
RATE_LIMITS_JSON_FILE	No	unset	/etc/pulp-engine/rate-limits.json	Path to a JSON file with the same shape — preferred for large configs or file-writing orchestration.
METRICS_TEMPLATE_LABEL_MODE	No	topn	allowlist	How templateKey is attached to render metrics: topn (LFU tracker admits up to METRICS_TEMPLATE_LABEL_MAX distinct keys, rest become __other__), allowlist (only listed keys emit real names — predictable, recommended for stable template sets), off (label always __other__). Bounds Prometheus label cardinality.
METRICS_TEMPLATE_LABEL_MAX	No	50	50	Cardinality cap for topn mode (max 10 000).
METRICS_TEMPLATE_LABEL_ALLOWLIST	No	unset	invoice,report	Comma-separated template keys for allowlist mode.
VARIANTS_ENABLED	No	false	true	Master switch for B.3 A/B variant label resolution. When false, labels with trafficSplit/controlLabel behave as plain pointers (no bucketing); when true, variant bucketing runs and renders carry X-PulpEngine-Bucket / X-PulpEngine-Resolved-Label headers.
RENDER_MAX_CONCURRENT_PAGES	No	5	16	Size of the Chromium page pool — simultaneous PDF renders per instance (v0.85.0; previously a compile-time constant of 5). Size against host CPU/RAM — budget ~150–300 MB per page under load; see the benchmark pack for measured per-slot throughput. In child-process mode the pool lives in the worker (the var is forwarded automatically). Range 1–50.
RENDER_MAX_QUEUE_DEPTH	No	2× pool size	10	Render-queue waiter cap (v0.85.0): renders beyond RENDER_MAX_CONCURRENT_PAGES + RENDER_MAX_QUEUE_DEPTH in flight are shed with 503 render_saturated + Retry-After instead of queueing unboundedly — graceful degradation instead of a timeout cascade. 0 = shed as soon as no page slot is free; a very large value approximates pre-v0.85.0 unbounded queueing. Range 0–1000.
RENDER_WORKER_TIMEOUT_MS	No	65000	65000	Per-phase render deadline (queue wait, then render) in child-process mode; the stream silence watchdog in container/socket modes (overall per-render ceiling derives as this + 55 s). Default = Puppeteer’s 30 s setContent + 30 s page.pdf + 5 s headroom, so Puppeteer surfaces its own typed timeout first. Raise for very large documents on slow hardware. Range 5 000–600 000.
RENDER_PREVIEW_RESERVED_SLOTS	No	1	1	Render slots of the RENDER_MAX_CONCURRENT_PAGES pool reserved exclusively for editor previews so batch/production load cannot starve interactive preview. 0 disables the reservation. Must leave at least one batch slot (≤ pool − 1; enforced at startup).
AUDIT_RETENTION_DAYS	No	unset (no automatic purge)	365	Opt-in scheduled purge of audit events older than this many days. Multi-instance: set on one instance only, or drive DELETE /audit-events from external cron. Range 1–3650.
AUDIT_PURGE_INTERVAL_HOURS	No	24	24	Interval between audit purge runs. Range 1–168.
BATCH_WEBHOOK_DLQ_RETENTION_DAYS	No	30	30	Batch-webhook DLQ rows older than this are deleted by an hourly purge. Rows typically become unreplayable after WEBHOOK_JOB_RETENTION_SECONDS but stay visible for audit until this retention. Range 1–365.
API_KEYS_DB	No	false	true	Enable DB API-key authentication without environment keys (PostgreSQL/SQL Server). Bootstrap requires an env admin key, an active DB admin key or a named admin user. Apply the API-key migration before deploying.
API_KEYS_CACHE_TTL_MS	No	10000	10000	Positive auth-cache lifetime, 1-60000 ms; unknown key IDs cache for 5 seconds. Management invalidates the local cache immediately. Other replicas converge within this bound.
API_JWT_ISSUERS_JSON	No	unset	[{"name":"service","issuer":"https://idp.example.com","audiences":["pulp-api"],"scopeMap":{"render":"render"},"tenant":{"fixed":"default"}}]	Configured API access-token issuers. Counts as credential intent. HTTPS required in production; asymmetric algorithms only. See OIDC guide.
API_JWT_ISSUERS_JSON_FILE	No	unset	/etc/pulp-engine/api-issuers.json	File containing the issuer array; mutually exclusive with the inline setting. Read once at boot; errors redact configuration contents.
API_KEYS_JSON_FILE	No	unset	/etc/pulp-engine/api-keys.json	Path to a file containing the API_KEYS_JSON array — preferred when credentials are mounted as files (Kubernetes secrets). Mutually exclusive with API_KEYS_JSON.
EDITOR_DIST_PATH	No	unset (Docker image: bundled editor)	../editor/dist	Path to a built editor bundle to serve at /editor/ (plus the form/embed wrapper routes). The Docker image sets this to its bundled editor; bare-metal source deployments point it at apps/editor/dist.
ANTHROPIC_TIMEOUT_MS	No	60000	60000	Timeout for each Anthropic HTTP call. Worst-case request latency is maxAttempts × ANTHROPIC_TIMEOUT_MS (a startup warning fires when that exceeds 5 minutes — reverse proxies tend to cut long requests).
AI_NODE_REWRITE_ENABLED	No	true	false	Dedicated switch for POST /nodes/rewrite (inline AI node rewrite). Only meaningful when ANTHROPIC_API_KEY is set; false disables node rewrite while leaving POST /templates/generate enabled.
RATE_LIMIT_AI_GENERATION_MAX	No	5	5	Per-actor per-minute limit for POST /templates/generate.
RATE_LIMIT_AI_EDIT_MAX	No	20	20	Per-identity per-minute limit for POST /nodes/rewrite (separate bucket from full-template generation).
RATE_LIMIT_AI_EDIT_OUTCOME_MAX	No	60	60	Per-identity per-minute limit for the rewrite apply/dismiss telemetry endpoint (own bucket so telemetry can’t burn rewrite quota).
PDF_QPDF_PATH	No	qpdf	qpdf	qpdf executable for AES-256 PDF protection. API images include it; local Windows installations must install qpdf separately. Missing tools return 503 and disable capabilities.pdf.protection.
PDF_PROTECT_TIMEOUT_MS	No	30000	30000	Protection subprocess timeout; range 100?300000 ms.
PDF_PROTECT_CONCURRENCY	No	4	4	Maximum simultaneous API-side protection processes; range 1?64.
PDF_UTIL_MAX_INPUT_SIZE_MB	No	50	50	Max size of a single input PDF for the /render/pdf/{merge,watermark,insert} transform endpoints. Range 1–500.
PDF_UTIL_MAX_TOTAL_SIZE_MB	No	200	200	Max combined input size per transform request. Range 1–1000.
PDF_UTIL_MAX_MERGE_SOURCES	No	50	50	Max PDFs in one merge request. Range 2–200.
PDF_UTIL_MAX_OUTPUT_PAGES	No	10000	10000	Max pages in a transform output PDF. Range 1–50 000.
RATE_LIMIT_PDF_UTIL_MAX	No	20	20	Per-minute rate limit for the PDF transform endpoints.
SANDBOX_ENABLED	No	false	true	Enables the anonymous public playground (/sandbox/*). Requires MULTI_TENANT_ENABLED=true and a hardened companion posture asserted at startup (non-in-process RENDER_MODE, BLOCK_REMOTE_RESOURCES=true, RATE_LIMIT_STORE=redis, TRUST_PROXY=true, concrete CORS_ALLOWED_ORIGINS, and SANDBOX_TOKEN_SECRET).
SANDBOX_TOKEN_SECRET	Yes (sandbox)	unset	<openssl rand -hex 32>	HMAC-SHA256 signing secret for sandbox session tokens (≥ 32 hex chars). Deliberately distinct from EDITOR_TOKEN_SECRET.
SANDBOX_TENANT_ID	No	sandbox	sandbox	Tenant slug sandbox sessions operate under (templates and templateRef resolution). Created/checked at boot; archiving it is the sandbox kill switch.
SANDBOX_SESSION_TTL_MINUTES	No	15	15	Sandbox session token lifetime. Range 1–240.
SANDBOX_SESSION_QUOTA	No	20	20	Renders allowed per session token (Redis-backed atomic counter). Range 1–500.
SANDBOX_BODY_LIMIT_BYTES	No	262144	262144	Body limit for sandbox render requests (smaller than PREVIEW_BODY_LIMIT — no base64-embedded content needed). Range 1 024–2 097 152.
SANDBOX_ALLOWED_ORIGINS	No	unset	https://pulpengine.dev	Extra CORS origins for /sandbox/*, unioned with CORS_ALLOWED_ORIGINS. Browser-side friction only — the real gate is the session token + IP rate limit.
RATE_LIMIT_SANDBOX_MAX	No	10	10	Per-IP per-minute limit across /sandbox/session and /sandbox/render/*.

Validation: config.ts reads and validates env vars at startup. The process exits immediately with a descriptive error if the variable required by the selected mode is missing — DATABASE_URL for postgres mode, SQL_SERVER_URL for sqlserver mode, TEMPLATES_DIR for file mode. For S3 mode, missing required S3 vars cause an immediate exit(1); an inaccessible bucket is detected at startup via a HeadBucket probe (fail-fast). Opt-in capability flags that depend on other variables (e.g. OIDC_DISCOVERY_URL without OIDC_CLIENT_ID) also fail fast.

---

Minimum Supported Production Configuration

When NODE_ENV=production, hardening is enforced by default — the server will not start unless all seven security controls are configured. Set HARDEN_PRODUCTION=false to explicitly opt out for evaluation. For supported production deployments, configure all seven controls:

When OIDC/SSO is enabled, two additional conditional controls are also enforced in hardened mode: OIDC_REDIRECT_URI must use https://, and OIDC_EDITOR_GROUPS must be set explicitly (leaving it at the default * would grant editor access to every authenticated SSO user, so hardened mode forces a deliberate choice).

# ── Minimum supported production configuration ──────────────────────────────────
CORS_ALLOWED_ORIGINS=https://editor.example.com   # specific origins; wildcard * rejected
DOCS_ENABLED=false                                 # disable Swagger UI (or set true to acknowledge exposure)
METRICS_TOKEN=<openssl rand -hex 32>               # bearer auth for GET /metrics
REQUIRE_HTTPS=true                                 # reject editor-token login over plain HTTP
TRUST_PROXY=true                                   # required when behind a TLS-terminating reverse proxy
BLOCK_REMOTE_RESOURCES=true                        # prevent render pipeline from fetching arbitrary external resources
EDITOR_USERS_JSON='[{"id":"admin","displayName":"Admin","key":"...","role":"admin"}]'
# or set ALLOW_SHARED_KEY_EDITOR=true to use shared-key identity
HARDEN_PRODUCTION=true                             # fail fast if any control above is missing

The server refuses to start until all seven controls pass (plus the conditional OIDC controls above, when OIDC is enabled). All violations are reported together in a single error, so you can fix everything in one pass.

Upgrading incrementally: If you need to configure controls one at a time, omit HARDEN_PRODUCTION — the advisory warning will list remaining gaps at startup. Set HARDEN_PRODUCTION=true once all seven are in place.

See § Hardened Production Mode for enforcement rules and the full example.

---

Object Storage (S3 / MinIO / R2)

Setting ASSET_BINARY_STORE=s3 stores uploaded asset binaries in an S3-compatible bucket instead of the local filesystem. This eliminates the shared-volume requirement for multi-instance deployments.

Required IAM / bucket permissions

The credentials (S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY) must have:

• Object-level access: s3:PutObject, s3:DeleteObject on objects in the bucket.
• Bucket-level access: s3:ListBucket (or equivalent) on the bucket itself — required for the HeadBucket startup probe. On AWS S3 this is documented in the HeadBucket API reference. For MinIO and other compatible providers, equivalent bucket-level access is required.

Public mode (default): The bucket and objects must be publicly readable at S3_PUBLIC_URL. Pulp Engine (and Puppeteer when rendering PDFs that reference assets) fetches asset URLs without auth headers.

Private mode (ASSET_ACCESS_MODE=private): The bucket does not need to be publicly readable. Pulp Engine fetches objects server-side using the configured credentials and proxies them to authorized callers. S3_PUBLIC_URL is not required in this mode — GetObject access on the bucket is sufficient. See § Asset Access Mode.

Public URL rules

Configuration	S3_PUBLIC_URL
Standard AWS S3 (no S3_ENDPOINT, no path-style)	Optional — auto-derived as https://{bucket}.s3.{region}.amazonaws.com
Custom endpoint (S3_ENDPOINT set)	Required — the API endpoint and public delivery URL differ for custom providers
Path-style URLs (S3_PATH_STYLE=true)	Required

Trailing slashes in S3_PUBLIC_URL are stripped automatically. Asset URLs are constructed as ${S3_PUBLIC_URL}/${filename}.

Standard AWS S3

ASSET_BINARY_STORE=s3
S3_BUCKET=my-pulp-engine-assets
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=AKIA...
S3_SECRET_ACCESS_KEY=...
# S3_PUBLIC_URL is optional — auto-derived as https://my-pulp-engine-assets.s3.us-east-1.amazonaws.com

MinIO (self-hosted)

ASSET_BINARY_STORE=s3
S3_BUCKET=pulp-engine-assets
S3_REGION=us-east-1          # arbitrary value — MinIO requires a non-empty region string
S3_ACCESS_KEY_ID=minio-access-key
S3_SECRET_ACCESS_KEY=minio-secret-key
S3_ENDPOINT=https://minio.example.com
S3_PATH_STYLE=true
S3_PUBLIC_URL=https://minio.example.com/pulp-engine-assets

Cloudflare R2

ASSET_BINARY_STORE=s3
S3_BUCKET=pulp-engine-assets
S3_REGION=auto
S3_ACCESS_KEY_ID=...
S3_SECRET_ACCESS_KEY=...
S3_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
S3_PUBLIC_URL=https://assets.example.com   # your R2 custom domain or public URL

Switching an existing deployment from filesystem to S3

Switching ASSET_BINARY_STORE from filesystem to s3 does not automatically migrate existing binaries. Existing AssetRecord.url values in the database or file index still point to /assets/{filename}. Without explicit migration those URLs become broken.

Options for existing deployments:

1. Greenfield cutover: If existing asset data is expendable, deploy fresh in S3 mode with an empty asset store. No migration needed.
2. Manual cutover: Copy existing binaries from ASSETS_DIR to the S3 bucket, update stored URL values in the database to the new S3 URLs, then switch to ASSET_BINARY_STORE=s3. Requires a maintenance window.

No automated migration tool is provided in this release.

---

Asset Access Mode

ASSET_ACCESS_MODE controls how uploaded image assets are delivered to the PDF renderer and the visual editor. Default: public.

public (default)

Assets are served without authentication:

• Filesystem: @fastify/static serves GET /assets/:filename without any X-Api-Key or session token required. Puppeteer fetches images via the loopback address (http://127.0.0.1:PORT/assets/...) during PDF rendering.
• S3: save() returns a public S3 URL (e.g. https://bucket.s3.region.amazonaws.com/filename). The bucket must be publicly readable. Puppeteer and the browser fetch images directly from S3/CDN.

private

All asset delivery is routed through an authenticated API proxy. No public URL is required.

• Filesystem: GET /assets/:filename requires X-Api-Key (admin or editor scope) or X-Editor-Token. @fastify/static is not registered.
• S3: save() returns a relative proxy URL (/assets/filename). The bucket does not need a public-read ACL or policy — only GetObject access for the API credentials is required. S3_PUBLIC_URL is not required.
• PDF rendering: Instead of Puppeteer fetching asset URLs at render time, the API inlines all referenced assets as base64 data URIs in the HTML before passing it to Puppeteer. No network calls from Puppeteer to the asset server are needed.
• HTML preview: Same inlining — the returned HTML contains self-contained data URIs rather than /assets/ src paths.
• Editor: The visual editor fetches /assets/:filename with X-Editor-Token and displays images via a revocable blob URL. No direct <img src> to /assets/ — auth is applied at fetch time.

Private mode configuration

ASSET_ACCESS_MODE=private

# Filesystem backend (no additional vars needed):
ASSET_BINARY_STORE=filesystem
ASSETS_DIR=/var/pulp-engine/assets
API_KEY_ADMIN=...
API_KEY_EDITOR=...

# S3 backend — bucket does not need to be public:
ASSET_BINARY_STORE=s3
S3_BUCKET=pulp-engine-private-assets
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=...
S3_SECRET_ACCESS_KEY=...
# S3_PUBLIC_URL is NOT required in private mode

IAM permissions for S3 private mode

In addition to s3:PutObject, s3:DeleteObject, and s3:ListBucket, the API credentials must have:

• s3:GetObject on objects in the bucket — required to stream binaries through the proxy handler.

Migration: switching from public to private mode

Filesystem: No data migration needed. Stored asset URLs are already in /assets/filename form. Switch ASSET_ACCESS_MODE=private and restart.

S3: Stored asset URLs from public mode are absolute S3 URLs (e.g. https://bucket.s3.region.amazonaws.com/filename). These URLs continue to work only as long as the S3 bucket remains publicly accessible. If you simultaneously privatize the bucket and set ASSET_ACCESS_MODE=private, those stored absolute URLs become inaccessible.

Options:

1. Keep bucket public during transition: Set ASSET_ACCESS_MODE=private, restart. Old assets continue to render via their stored public URLs. New uploads get proxy URLs (/assets/filename). Over time, as old assets are replaced with newly uploaded ones, all stored URLs become proxy URLs.
2. Hard cutover: Simultaneously privatize the bucket and set ASSET_ACCESS_MODE=private. Re-upload all assets (which generates proxy URLs) or rewrite stored URL values in your database from absolute S3 URLs to /assets/filename. Requires a maintenance window.

No automated migration tool is provided.

---

Docker Deployment (recommended)

Three images are published to GitHub Container Registry on every tagged release:

Image	Purpose
ghcr.io/OWNER/pulp-engine:vX.Y.Z	API server — all storage modes, all render modes; includes bundled editor SPA
ghcr.io/OWNER/pulp-engine-worker:vX.Y.Z	Ephemeral PDF render worker — used by RENDER_MODE=container and RENDER_MODE=socket
ghcr.io/OWNER/pulp-engine-controller:vX.Y.Z	Render controller — sole Docker socket holder in RENDER_MODE=socket topology

Replace OWNER with your GitHub organisation or the org that published your release. All images are tagged with the version (e.g. v0.48.1) and a rolling latest.

Most deployments only need the API image. Pull the worker image when using RENDER_MODE=container or RENDER_MODE=socket; pull the controller image as well only for RENDER_MODE=socket, the privilege-separated topology (see § Render Isolation Mode).

Size: pulp-engine ~600–800 MB (Chromium binary); pulp-engine-worker similar (also needs Chromium); pulp-engine-controller ~200 MB (Docker CLI only, no browser)

Quick start — file mode (no database required)

docker pull ghcr.io/OWNER/pulp-engine:vX.Y.Z

docker run -d \
  --name pulp-engine \
  -p 3000:3000 \
  -e API_KEY_ADMIN=your-secret-key \
  -e STORAGE_MODE=file \
  -v /var/pulp-engine/templates:/data/templates \
  -v /var/pulp-engine/assets:/data/assets \
  ghcr.io/OWNER/pulp-engine:vX.Y.Z

Templates volume must be writable. File mode writes template definitions and version snapshots under TEMPLATES_DIR (/data/templates in the image) on every save and publish. Do not add :ro to that mount.

PostgreSQL mode — with migration

Migrations must run before the app container starts. Use the same image tag to ensure code and schema are always in sync:

# 1. Run migrations (one-off container — exits after apply)
docker run --rm \
  --entrypoint /app/node_modules/.bin/prisma \
  -e DATABASE_URL=postgres://user:pass@host:5432/pulp-engine \
  ghcr.io/OWNER/pulp-engine:vX.Y.Z \
  migrate deploy --schema /app/src/prisma/schema.prisma

# 2. Start the app container
docker run -d \
  --name pulp-engine \
  -p 3000:3000 \
  -e NODE_ENV=production \
  -e STORAGE_MODE=postgres \
  -e DATABASE_URL=postgres://user:pass@host:5432/pulp-engine \
  -e API_KEY_ADMIN=your-secret-key \
  -v /var/pulp-engine/assets:/data/assets \
  ghcr.io/OWNER/pulp-engine:vX.Y.Z

Already-applied migrations are skipped; this is safe to re-run.

Container environment variables

All variables from § 2 Environment Variables can be passed with -e. The following defaults are baked into the image:

Variable	Image default	Override when
NODE_ENV	production	Never — leave as-is
HOST	0.0.0.0	Using HOST=127.0.0.1 with a sidecar proxy
PORT	3000	Changing the listener port
STORAGE_MODE	file	Using postgres or sqlserver
ASSETS_DIR	/data/assets	Mounting assets at a different path
PULP_ENGINE_DISABLE_SANDBOX	true	Never — required for containers; do not unset

Volume mounts

Path in container	Purpose	Mount a host path?
/data/assets	Uploaded image assets	Yes — persist across container restarts
/data/templates	Template definitions (file mode only)	Yes — read-write; the API writes template files on every save and publish

Deployment validation

Two scripts verify a running deployment. Use them in order after starting the container.

validate-deploy.sh — infrastructure and security posture. Checks liveness, readiness, metrics endpoint, security advisories (open /metrics, exposed Swagger UI), auth, and optionally a stored-template render. Requires only curl. See the script header for full argument and check descriptions.

# Infrastructure + auth (no seeded templates required)
./scripts/validate-deploy.sh http://localhost:3000 $API_KEY_ADMIN

# After seeding or creating a template — adds a render pipeline check
./scripts/validate-deploy.sh http://localhost:3000 $API_KEY_ADMIN loan-approval-letter $METRICS_TOKEN

# Docker image deployments — also verify the bundled editor SPA
EXPECT_EDITOR=true ./scripts/validate-deploy.sh http://localhost:3000 $API_KEY_ADMIN

smoke-test.sh — end-to-end lifecycle. Creates a disposable template, renders it to PDF, verifies the editor route, and cleans up. Requires curl and jq.

./scripts/smoke-test.sh http://localhost:3000 $API_KEY_ADMIN

Recommended first-deployment sequence:

1. Start the container (or bare-metal process).
2. Run validate-deploy.sh — confirms health, storage, metrics, and auth are working.
3. Run smoke-test.sh — confirms the full create → render → delete lifecycle.
4. If HARDEN_PRODUCTION=true: verify 0 warnings from validate-deploy.sh.

SQL Server parity note: CI exercises SQL Server schema migration and SQL Server-specific storage/API tests in the test-sqlserver job (ci.yml). Local deployment rehearsal for SQL Server requires a reachable SQL Server instance — follow the SQL Server mode deployment steps and run the same validation scripts above.

Rollback

# Stop current container
docker stop pulp-engine && docker rm pulp-engine

# Start the previous version
docker run -d --name pulp-engine [same -e and -v flags] \
  ghcr.io/OWNER/pulp-engine:v0.PREV.Y

# Validate the rollback
./scripts/validate-deploy.sh http://localhost:3000 $API_KEY_ADMIN

# If rolling back a postgres migration:
# The previous image's migrations were already applied — no migration rollback is
# needed unless you added backward-incompatible schema changes. In that case,
# restore from a database backup taken before the forward migration was applied.
#
# If rolling back a SQL Server migration:
# Same principle — no migration rollback command exists. Restore from a backup taken
# before the forward migration was applied if schema changes were backward-incompatible.

Chrome sandbox note

PULP_ENGINE_DISABLE_SANDBOX=true is set automatically in the image. This is required because Docker containers run without the kernel-level sandbox that Chrome requires. The container network boundary and Puppeteer’s request-interception layer provide equivalent containment for the threat model of this service. Do not unset this variable in containerised deployments.

SQL Server mode — with migration

Migrations must run before the app container starts. Use the same image tag to ensure code and schema are always in sync:

# 1. Run migrations (one-off container — exits after apply)
docker run --rm \
  --entrypoint node \
  -e SQL_SERVER_URL=mssql://user:pass@host:1433/pulp-engine?trustServerCertificate=true \
  ghcr.io/OWNER/pulp-engine:v0.X.Y \
  /app/dist/scripts/migrate-sqlserver.js

# 2. Start the app container
docker run -d \
  --name pulp-engine \
  -p 3000:3000 \
  -e NODE_ENV=production \
  -e STORAGE_MODE=sqlserver \
  -e SQL_SERVER_URL=mssql://user:pass@host:1433/pulp-engine?trustServerCertificate=true \
  -e API_KEY_ADMIN=your-secret-key \
  -v /var/pulp-engine/assets:/data/assets \
  ghcr.io/OWNER/pulp-engine:v0.X.Y

Already-applied migrations are skipped; this is safe to re-run. The runner connects to master, creates the target database if absent, then applies all pending .sql files from dist/storage/sqlserver/migrations/ in order. Applied migrations are tracked in dbo.__migrations.

For repo-checkout deployments (non-Docker), continue using:

pnpm --filter @pulp-engine/api db:migrate:sqlserver

Async job durability across storage backends

PostgreSQL and SQL Server both persist job metadata and private filesystem/S3 result bytes. Apply the new render-job and idempotency migrations before starting an upgraded API. Completed and cancelled jobs, including partial results, survive restart until retention. Automatic worker resumption is not implemented: interrupted active work follows the startup orphan policy. Shared volumes or S3 are required across replicas.

BATCH_ASYNC_DURABILITY operator gate

This tri-state gate applies to file mode only. required refuses boot with FILE_BATCH_NONDURABLE; warn logs the limitation; allow-nondurable accepts process-local jobs quietly. Hardened production defaults to required, other deployments to warn. SQL Server now satisfies the durable metadata requirement, like PostgreSQL.

The retained WEBHOOK_* settings control admission, job retention, job timeout and optional callback retries for unified jobs too. Admission is per API instance. Idempotency defaults to shared rate-limit Redis when present, otherwise the configured database, otherwise process-local memory. An explicit unavailable store fails startup. The hourly purge removes expired records. Set lock TTL above the longest request that can perform a durable side effect; stale locks can be reacquired and lease fencing prevents an older response from overwriting the new lease.

---

Visual Editor

The visual editor is bundled in the Docker image and served at /editor/. No separate hosting or deployment is required.

Access: http://[your-host]:3000/editor/ (or https://pulp-engine.example.com/editor/ behind a reverse proxy). Navigating to /editor (without trailing slash) redirects automatically.

Authentication: The editor shows a login screen on first load. Enter the value of API_KEY_EDITOR (or API_KEY_ADMIN) to obtain a session token. See docs/editor-guide.md for the full auth model.

Live preview

The editor’s preview button calls POST /render/preview/html. This route is not registered by default in production — it returns 404 unless you explicitly set:

PREVIEW_ROUTES_ENABLED=true

The compose evaluator files (compose.yaml, compose.postgres.yaml) set this automatically. For production, weigh the exposure: preview routes accept an inline TemplateDefinition and render it server-side. Pair with network-level restrictions if enabled. See the PREVIEW_ROUTES_ENABLED env var entry in § 2 Environment Variables for full details.

Without PREVIEW_ROUTES_ENABLED=true: template load, save, publish, version history, and asset management all work — only the in-editor live preview is unavailable.

CORS and same-origin deployments

The bundled editor makes same-origin requests to the API (editor at /editor/, API at /). The browser does not send cross-origin headers for same-origin requests — CORS_ALLOWED_ORIGINS is not required for the editor to reach the API in this topology.

Set CORS_ALLOWED_ORIGINS when:

• other browser clients or SPAs on different origins need API access, or
• HARDEN_PRODUCTION=true is set (the hardened startup check requires the var regardless — set it to your deployment URL, e.g. CORS_ALLOWED_ORIGINS=https://pulp-engine.example.com).

Local development vs bundled deployment

Context	Editor URL	API URL	Notes
pnpm dev (source)	http://localhost:5174	http://localhost:3000	Vite dev server; hot reload
Docker image	http://[host]:3000/editor/	http://[host]:3000	Static SPA served by Fastify; same origin

---

Multi-Tenant Mode (opt-in)

MULTI_TENANT_ENABLED=true turns on per-tenant isolation of templates, assets, credentials, audit events, and scheduled jobs.

Requirement	Value
STORAGE_MODE	Must be postgres or sqlserver. File mode is rejected at startup when multi-tenant is enabled.
API_KEYS_JSON	Replaces the single API_KEY_* variables. Each entry carries tenantId + scopes; a super-admin entry (tenantId: null) is required for /admin/tenants CRUD.
X-Editor-Token	Carries a signed tenantId claim — cross-tenant access is rejected.
Asset binary store	S3 prefixes and filesystem paths are tenant-scoped; no operator action required beyond enabling multi-tenant.
Plugin storage	Plugin data is tenant-scoped; plugins that cannot honour tenantId are rejected at registration.

See tenant-isolation-guarantees.md for the full isolation model and the small set of operational caveats (e.g. shared Chromium render pool, shared rate-limit buckets unless configured otherwise).

Shared compute boundary — choose your render mode deliberately. Isolation in multi-tenant mode is row/path-level (templates, assets, credentials, audit, schedules), not compute-level. In the default RENDER_MODE=child-process, all tenants share one Chromium render pool per API pod, so a bug in a Handlebars helper or the render pipeline could in principle leak state across tenants within a pod. This is acceptable for the product’s trusted-tenant model (credentialled internal authors). If your tenants are untrusted or semi-trusted, run RENDER_MODE=container or socket for per-render compute isolation — see § Render Isolation Mode and the “Shared compute” caveat in tenant-isolation-guarantees.md.

---

Render Isolation Mode

RENDER_MODE controls where Puppeteer runs during PDF generation. Choose based on your security requirements and whether Docker is available.

For the security-reviewer view of these modes — threat boundaries, residual risks, operator checklist per mode — see Render Isolation Threat Model. This section covers operational setup; the threat-model doc covers what each mode actually isolates and what survives an API compromise.

Mode	Docker required	API holds Docker socket	Isolation level
child-process (default)	No	—	Process isolation, allowlisted worker env (no API/storage secrets)
container	Yes	Yes	Container: network-none, read-only FS, cap-drop ALL
socket	Yes (on controller)	No	Container (same flags) + privilege separation
in-process	No	—	None — debugging only

Sizing the render pool (v0.85.0)

PDF capacity per instance is RENDER_MAX_CONCURRENT_PAGES simultaneous Chromium pages (default 5, minus RENDER_PREVIEW_RESERVED_SLOTS for the batch lane). Renders beyond pool + RENDER_MAX_QUEUE_DEPTH in flight are shed with 503 render_saturated + Retry-After: 30 — clients should back off and retry rather than treating it as a render failure. The benchmark pack carries measured per-slot throughput and the saturation arithmetic (its reference rig sustained 16 concurrent renders host-CPU-bound); use it to pick a pool size, then budget ~150–300 MB RAM per page under load and watch the pulp_engine_render_dispatches_in_flight gauge — pegged at the cap with 503s in the logs means scale the pool or add replicas. Render capacity scales linearly with replica count behind a load balancer (see HA reference architecture).

child-process (default)

Puppeteer runs in a persistent child process spawned with a minimal allowlisted worker environment — no API, database, or object-store credentials cross the boundary. See Render Isolation Threat Model § 3 for the exact allowlist. Recommended for environments without Docker.

No additional configuration needed.

container

Puppeteer runs in a fresh ephemeral Docker container per render. Provides strong isolation but the API process holds Docker socket authority. A compromised API process could invoke arbitrary Docker operations on the host.

Required environment:

RENDER_MODE=container
RENDER_CONTAINER_IMAGE=ghcr.io/OWNER/pulp-engine-worker:vX.Y.Z

Build the worker image:

docker build -f Dockerfile.worker -t ghcr.io/OWNER/pulp-engine-worker:vX.Y.Z .

Runtime flags applied per render (hardcoded in the dispatcher):

• --network none — no network access from worker
• --read-only — read-only root filesystem
• --tmpfs /tmp:rw,noexec,nosuid,size=256m — ephemeral writable /tmp
• --cap-drop ALL — no Linux capabilities
• --security-opt no-new-privileges — no setuid escalation
• --memory 512m --cpus 1 — resource limits (configurable)
• --rm — container removed after each render

socket (recommended for container mode in production)

The API delegates Docker invocations to a separate render-controller process over a Unix domain socket. The API process holds no Docker socket authority. Even if the API is compromised, it cannot invoke arbitrary Docker operations — it can only submit render requests through the narrow socket protocol.

The controller enforces the same hardcoded security flags regardless of what the API requests.

Required environment (API process):

RENDER_MODE=socket
RENDER_CONTROLLER_SOCKET=/run/render/render.sock

Required environment (controller process):

RENDER_CONTAINER_IMAGE=ghcr.io/OWNER/pulp-engine-worker:vX.Y.Z
RENDER_CONTROLLER_SOCKET=/run/render/render.sock
# Optional:
# RENDER_CONTAINER_MEMORY_LIMIT=512m
# RENDER_CONTAINER_CPU_LIMIT=1
# RENDER_CONTAINER_SECCOMP_PROFILE=/etc/pulp-engine/chromium-seccomp.json

Get the images. All three (pulp-engine, pulp-engine-worker, pulp-engine-controller) are published publicly to GHCR on every release, so the simplest path is to pull:

docker pull ghcr.io/troycoderboy/pulp-engine-worker:vX.Y.Z
docker pull ghcr.io/troycoderboy/pulp-engine-controller:vX.Y.Z

Or build them from source if you need a local/custom build:

docker build -f Dockerfile.worker -t ghcr.io/OWNER/pulp-engine-worker:vX.Y.Z .
docker build -f Dockerfile.controller -t ghcr.io/OWNER/pulp-engine-controller:vX.Y.Z .

Deploy with the provided compose file, which shows the correct privilege separation:

docker compose -f compose.container.yaml up -d

The compose file:

• Mounts /var/run/docker.sock only on the render-controller service
• Uses condition: service_healthy so the API waits until the controller socket is live
• Routes all renders through the socket; the API container has no Docker mount

Optional seccomp profile: Set RENDER_CONTAINER_SECCOMP_PROFILE to the path of a seccomp JSON file inside the controller container. The profile is passed as --security-opt seccomp=<path> to each worker container. You must bind-mount the file into the controller container:

render-controller:
  volumes:
    - /etc/pulp-engine/chromium-seccomp.json:/etc/pulp-engine/chromium-seccomp.json:ro
  environment:
    RENDER_CONTAINER_SECCOMP_PROFILE: /etc/pulp-engine/chromium-seccomp.json

Security model and residual risks

For the threat-radius comparison across modes, what survives an API compromise, the durable identity boundary (UIDs, socket mode 0660), and the per-mode operator checklist, see Render Isolation Threat Model.

---

HTML conversion and inline definitions

Variable	Default	Behavior
CONVERT_HTML_ENABLED	false	Enable POST /render/convert/pdf. Raw HTML is trusted author input; JavaScript remains disabled.
CONVERT_BODY_LIMIT	2097152	Maximum conversion request bytes.
RATE_LIMIT_CONVERT_MAX	20	Default conversion requests per rate-limit window; independent convert route class.
CONVERT_URL_ALLOWED_ORIGINS	unset	Comma-separated exact HTTP(S) origins; URL conversion is disabled when empty. Every redirect is checked, and private addresses stay blocked.
CONVERT_URL_MAX_BYTES	5242880	Maximum upstream HTML response bytes.
CONVERT_URL_TIMEOUT_MS	10000	Whole URL conversion fetch timeout in milliseconds, including redirects and body reads.
INLINE_TEMPLATES_ENABLED	false	Allow production render credentials to supply inline definitions. Stored-key/label-restricted keys remain unable to author arbitrary definitions.
INLINE_TEMPLATE_MAX_BYTES	524288	Maximum serialized inline definition bytes, shared across a batch.
INLINE_TEMPLATES_ALLOW_DATA_SOURCES	false	Permit inline templateRef data sources; failures use production semantics.

Production requests continue to use RENDER_BODY_LIMIT_BYTES and batches use BATCH_BODY_LIMIT_BYTES; those existing settings are reused. In hardened mode HTML conversion requires an isolated RENDER_MODE and BLOCK_REMOTE_RESOURCES=true. Roll API and PDF worker images together so disableJavaScript is enforced in every worker. Inline author trust extends to unrestricted render credentials when enabled; use container or socket isolation for untrusted authors.

3. Database Setup

Database modes (STORAGE_MODE=postgres or sqlserver). Skip this section entirely if using file mode — no migration or seed step is required.

Postgres mode

# 1. Create the database (if not already created by your DBA / managed service)
createdb -U postgres pulp-engine

# 2. Apply all migrations
pnpm --filter @pulp-engine/api db:deploy

# 3. Verify both tables exist
psql -U postgres -d pulp-engine -c "\dt"
# Should show: templates, template_versions

pnpm db:deploy runs prisma migrate deploy, which applies every committed migration in order. This is safe to re-run — already-applied migrations are skipped.

SQL Server mode

# 1. Apply the schema (creates the database if absent; idempotent)
pnpm --filter @pulp-engine/api db:migrate:sqlserver

# 2. Verify the tables exist (optional)
sqlcmd -S <host> -d <database> -U <user> -P <pass> \
  -Q "SELECT name FROM sys.tables ORDER BY name" -C
# Expected: assets, template_versions, templates

The migration runner connects to master, creates the target database if absent, then applies all pending migration files from apps/api/src/storage/sqlserver/migrations/ in order (e.g. 001_init.sql, 002_add_created_by.sql). Applied migrations are tracked in dbo.__migrations — already-applied files are skipped. Each migration runs in a transaction; if one fails the transaction is rolled back and the script exits non-zero naming the failing file. Safe to re-run at any time.

Seed sample templates

pnpm db:seed

This loads loan-approval-letter and sample-invoice into the configured store (postgres, sqlserver, or file mode). Idempotent — safe to re-run.

Prisma client generation

The Prisma client must be generated from the schema before building:

pnpm db:generate

This is a one-time step per deployment machine. The generated client is written to node_modules — it is not committed to source control.

---

4. Build and Start

Production build

# From repo root — builds all packages in dependency order via Turborepo
pnpm build

Output is compiled TypeScript written to each package’s dist/ directory.

Start the API

node apps/api/dist/index.js

Or, via pnpm filter:

pnpm --filter @pulp-engine/api start

Confirm the process is running:

curl http://localhost:3000/health
# { "status": "ok", "timestamp": "..." }

Process management

Use a process manager to keep the API running across crashes and server restarts:

# PM2 example
pm2 start apps/api/dist/index.js --name pulp-engine-api
pm2 save
pm2 startup   # generates a startup script for your init system

Alternatively, create a systemd unit file pointing to node apps/api/dist/index.js.

---

5. Reverse Proxy and Network Placement

For internal use, deploy behind a reverse proxy on a private network. The API requires an X-Api-Key header in production — set at least API_KEY_ADMIN (and optionally API_KEY_RENDER / API_KEY_PREVIEW / API_KEY_EDITOR) in the environment. Even with auth, do not expose port 3000 directly to the internet.

Recommended network topology

[Internal callers] → [nginx / IIS / API Gateway] → [Pulp Engine API :3000]

nginx example

server {
    listen 80;
    server_name pulp-engine.internal;

    location / {
        proxy_pass         http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;

        # PDF responses can be large; allow adequate timeout and buffer
        proxy_read_timeout     60s;
        proxy_buffers          16 256k;
        proxy_buffer_size      256k;
    }
}

Bind Pulp Engine itself to HOST=127.0.0.1 so it only accepts connections from the reverse proxy, not directly from the network.

HTTPS (TLS-terminating proxy) example

For production deployments with HARDEN_PRODUCTION=true, TLS termination at the reverse proxy is the expected topology. The proxy handles certificates and forwards X-Forwarded-Proto so Pulp Engine can enforce HTTPS on sensitive routes.

# Redirect HTTP → HTTPS
server {
    listen 80;
    server_name pulp-engine.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name pulp-engine.example.com;

    ssl_certificate     /etc/nginx/ssl/pulp-engine.crt;
    ssl_certificate_key /etc/nginx/ssl/pulp-engine.key;

    location / {
        proxy_pass         http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;

        # PDF responses can be large; allow adequate timeout and buffer
        proxy_read_timeout     60s;
        proxy_buffers          16 256k;
        proxy_buffer_size      256k;
    }
}

Set the following in the Pulp Engine .env to match this topology:

TRUST_PROXY=true       # Fastify reads X-Forwarded-Proto from the proxy
REQUIRE_HTTPS=true     # rejects editor-token login over plain HTTP
HOST=127.0.0.1         # accept connections from the proxy only

For the bundled same-origin editor deployment (/editor/ on the same host), CORS_ALLOWED_ORIGINS is not required for the editor to reach the API — the browser treats these as same-origin requests. However, HARDEN_PRODUCTION=true requires the variable regardless; set it to your deployment URL (e.g. CORS_ALLOWED_ORIGINS=https://pulp-engine.example.com). HTTPS proxying and CORS configuration are independent concerns. See § Hardened Production Mode for the full hardened .env example.

Common mistake: Setting REQUIRE_HTTPS=true without TRUST_PROXY=true causes editor-token login requests to fail. Behind a TLS-terminating proxy the internal socket is plain HTTP; without TRUST_PROXY=true, Fastify cannot read X-Forwarded-Proto and sees every request as non-HTTPS.

---

6. Logging and Operational Visibility

Log format

Fastify uses pino for structured logging.

NODE_ENV	Log format	Default level
production	JSON (one object per line)	info
development / anything else	Pretty-printed	info

Set NODE_ENV=production in deployed environments so log aggregators (Datadog, Splunk, CloudWatch, etc.) can parse lines as JSON. Adjust LOG_LEVEL to control verbosity.

Key log fields

Field	Description
level	info, warn, error
time	Unix milliseconds
reqId	Per-request ID assigned by Fastify
req.method, req.url	Route context
res.statusCode, responseTime	Response details
err.message, err.stack	Error context on failures
reason	Bounded failure reason — invalid_key, missing_key, insufficient_scope, invalid_token, template_not_found, render_error, version_conflict, duplicate_key, not_found
source	production or preview — render origin
outcome	success or failure — operation result
event	Audit event name — editor_token_minted, template_mutation, asset_mutation (v0.20.0+)
actor	Operator-supplied actor label from the session token, or null (v0.20.0+). Present on audit events only
credentialScope	admin or editor — which credential scope performed the write. Present on audit events only

Audit log events (v0.20.0+): Template and asset write operations (POST /templates, PUT /templates/:key, DELETE /templates/:key, restore, asset upload, asset delete) emit structured log entries with event, operation, actor, credentialScope, and a resource identifier (templateKey or assetId). These fields are intentionally higher-cardinality than operational metrics and are designed for log aggregation queries — not Prometheus metric labels. actor: null means the write was performed via direct X-Api-Key auth or no actor label was supplied at login.

Cluster-upgrade note (v0.20.0): In multi-instance deployments, upgrade all API instances to v0.20.0 before operators begin supplying actor labels at login. Using the 3-part token format (omitting actor in the login form) during a mixed-version rollout window is safe.

Health endpoints

SHUTDOWN_DRAIN_DELAY_MS defaults to 0 (range 0-300000 ms). On SIGTERM the API marks readiness draining, waits this delay, then closes. The Helm chart uses 10000. Allow enough container termination grace for preStop, this delay and the render-worker timeout. Use /health for startup/liveness and /health/ready for readiness; startup must allow Chromium warmup. Draining returns 503 with draining: true while liveness stays healthy.

GET /health — liveness probe. Always returns 200 { "status": "ok", "version": "0.51.0", "timestamp": "..." } if the process is running. No dependency checks. Use as a load-balancer liveness check.

GET /health/ready — readiness probe. Verifies that storage, asset binary store, and renderer are each reachable within 2 seconds (checks run in parallel). In API-only mode (no render dispatcher, preview disabled), the renderer check always reports "ok".

• 200 { "status": "ok", "version": "...", "checks": { "storage": "ok", "assetBinaryStore": "ok", "renderer": "ok" } } — all subsystems reachable
• 503 { "status": "degraded", "version": "...", "checks": { "storage": "error", "assetBinaryStore": "ok", "renderer": "ok" } } — one or more subsystems unreachable

Use GET /health/ready as your Kubernetes readinessProbe to prevent traffic from reaching the pod before all subsystems are available, and to drain traffic during outages. A 503 means the API is draining or at least one check returned "error" or "timeout": storage — template/metadata store unreachable (check database connectivity or file system access); assetBinaryStore — binary asset store unreachable (check file system or S3 connectivity); renderer — Chromium browser or render dispatcher unresponsive (check browser process or container worker). Any single failing subsystem causes 503; validate-deploy.sh treats this as a hard failure.

Preview capability — GET /render/preview/status

Returns the cached startup capability state of the preview renderer. This endpoint is always registered (regardless of PREVIEW_ROUTES_ENABLED) and requires editor, preview, or admin credentials.

{ "available": true, "reason": null }
{ "available": false, "reason": "routes_disabled" }
{ "available": false, "reason": "browser_unavailable" }

Always returns HTTP 200. The available field is the signal; reason describes why when unavailable.

reason	Meaning	Action
(absent / null)	Preview is available. Chromium launched successfully at startup.	—
routes_disabled	PREVIEW_ROUTES_ENABLED is not set; preview render routes are not registered.	Set PREVIEW_ROUTES_ENABLED=true to enable, or leave disabled (default and recommended production posture).
browser_unavailable	Chromium failed to launch at startup. Routes are registered but cannot serve requests.	Check system libraries (see § 1 Runtime Requirements). Check the API startup logs for the raw browser error. Set PULP_ENGINE_DISABLE_SANDBOX=true if running in a container.

Important: this is a startup snapshot, not a perpetual guarantee. available: true means Chromium launched when the process started. Individual render failures after startup (bad template, out-of-memory, etc.) travel the normal 422/500 response path and are not reflected here.

Diagnostic use:

curl -H "X-Api-Key: $ADMIN_KEY" http://localhost:3000/render/preview/status

Audit events — GET /audit-events

All template mutations, asset mutations, and editor token mints are persisted to the same database as templates and assets (no additional infrastructure required). Query the audit trail with GET /audit-events (admin scope required). See the API guide for filter parameters.

Audit events can be purged via DELETE /audit-events?before=<ISO 8601> (admin scope). See the runbook for retention cron examples.

Metrics — GET /metrics

Returns Prometheus text format 0.0.4. Rate limiting disabled. When METRICS_TOKEN is set, requires Authorization: Bearer <token> — see § Environment Variables. When METRICS_TOKEN is not set, the endpoint is unauthenticated (backward-compatible default). In either case, restrict access at the network layer in production.

Scrape config example (Prometheus / Grafana Agent):

scrape_configs:
  - job_name: pulp-engine
    static_configs:
      - targets: ['localhost:3000']
    metrics_path: /metrics

Key application metrics:

Metric	What to alert on
pulp_engine_render_requests_total{status="failure"}	Render failure rate spike
pulp_engine_auth_failures_total{reason="invalid_key"}	Potential credential scanning
pulp_engine_template_mutations_total{status="conflict"}	Elevated optimistic-concurrency conflicts
pulp_engine_http_request_duration_seconds{route="render_pdf"}	PDF render latency SLO

Network access: /metrics is not secret but is intended for internal scrapers only. Restrict at the network layer:

# nginx — allow scraper IP, deny all others
location /metrics {
    allow 10.0.0.50;   # Prometheus / Grafana Agent IP
    deny  all;
    proxy_pass http://127.0.0.1:3000;
}

---

7. Backup and Recovery

Postgres mode

Templates are stored as JSONB in PostgreSQL (template_versions.definition).

What to back up

The entire database. Pulp Engine’s durable state spans 14 tables — alongside templates / template_versions / assets, the database holds the tenant registry, editor users, template labels and sample data, audit events, render usage, schedules + executions + their delivery DLQ, and async-batch jobs + their webhook DLQ. A table-filtered dump silently drops scheduled deliveries, audit history, user identities, and label promotions — restore from one and they are gone.

Item	Contents	Criticality
Full database (pg_dump, no --table filters)	All 14 durable tables — see schema.prisma	High
ASSETS_DIR filesystem directory (or the S3 bucket)	Uploaded image binaries	Medium — loss means broken image refs in PDFs

Note: The database and the asset binaries must be backed up together — an assets record without its file will produce broken image references, and a file without a record will be orphaned.

Recommended backup strategy

# Daily full-database logical backup (custom format, restorable via pg_restore)
pg_dump --format=custom --no-owner --no-privileges \
  --file="pulp-engine-$(date -u +%Y%m%dT%H%M%SZ).dump" \
  "$DATABASE_URL"

# Sync asset files to a backup location
rsync -a /var/pulp-engine/assets/ /backups/pulp-engine-assets/

Restore: follow the backup & restore runbook § 3 — the canonical sequence is stop API → restore binaries → drop/recreate the database (quoted identifiers) → pg_restore → prisma migrate deploy → start API → verify. Restoring into a live database with the API running is not supported. The sequence is rehearsed in CI by scripts/restore-rehearsal.sh.

Source-of-truth for templates

The canonical template definitions are the JSON files in templates/ in the repository. If the database is lost, re-run pnpm db:seed to restore the seed templates. Custom templates added post-seed are only in the database — back them up via pg_dump.

Recommendation: Before adding custom templates through the API in production, export each template via GET /templates/:key and commit the JSON to source control.

SQL Server mode

Templates are stored in the template_versions.definition column (NVARCHAR(MAX) JSON).

What to back up

Same principle as postgres mode: the full database (the T-SQL BACKUP DATABASE below already is full-database), plus ASSETS_DIR.

Recommended strategy

# Full database backup (T-SQL, run on the SQL Server host or via sqlcmd)
BACKUP DATABASE [pulp-engine] TO DISK = '/backups/pulp-engine.bak' WITH FORMAT, COMPRESSION;

# Sync asset files
rsync -a /var/pulp-engine/assets/ /backups/pulp-engine-assets/

For Azure SQL, use the built-in automated backup (configurable retention) plus blob storage for asset files.

Source-of-truth for templates

Same as postgres mode: canonical definitions are in templates/ in the repository. Custom templates created post-seed are only in the database — export via GET /templates/:key and commit to source control.

File mode

Template state lives in TEMPLATES_DIR (one folder per template key). The JSON files in that directory are the store — edits made through the API write back to TEMPLATES_DIR automatically; no seed step is required.

Back up TEMPLATES_DIR like any other important filesystem data:

rsync -a /var/pulp-engine/templates/ /backups/pulp-engine-templates/

ASSETS_DIR must be backed up in both binary modes: it always holds the asset metadata index (.assets-index.json), and when ASSET_BINARY_STORE=filesystem (the default) it holds the binaries too. When ASSET_BINARY_STORE=s3, binaries live in your S3 bucket — back the bucket up per the runbook § 2.2, and still back up ASSETS_DIR for the index.

Note: File mode is designed for single-instance deployments. Concurrent writes from multiple API processes to the same TEMPLATES_DIR are not supported.

---

8. Credential Migration (API_KEY → scoped keys)

If you have an existing deployment using the single API_KEY, follow these steps to migrate to the scoped credential model without downtime.

Steps

1. Generate new secrets for each scope you need:

   # Example: generate with openssl
   openssl rand -hex 32   # for API_KEY_ADMIN
   openssl rand -hex 32   # for API_KEY_RENDER (optional — only if needed)

2. Update .env to use the new keys. Do not set API_KEY alongside the new keys — the server rejects ambiguous configs at startup:

   # Before:
   API_KEY=my-old-secret
   
   # After:
   API_KEY_ADMIN=my-new-admin-secret
   API_KEY_RENDER=my-new-render-secret   # optional

3. Update callers to send the appropriate scoped key:

   • Integrations that only render documents → use API_KEY_RENDER
   • Visual editor → set API_KEY_EDITOR on the API server; operators enter this value in the editor’s login form (no frontend env var required)
   • Everything else → use API_KEY_ADMIN

4. Restart the API process and verify startup completes without errors.

5. Verify each integration still works by checking a health check then a representative request.

If you need a safe migration window

The legacy API_KEY is still accepted (treated as admin scope) with a deprecation warning logged at startup. You can leave it in place while updating callers, then remove it once all callers have been switched. You cannot run both the old key and the new keys simultaneously — that combination is rejected.

---

Named-User Mode (EDITOR_USERS_JSON)

v0.23.0+ — applies only to the visual editor login gate. Programmatic X-Api-Key access is unaffected.

Named-user mode replaces the shared API_KEY_EDITOR login with per-user personal keys. Each team member has a unique credential, a verified identity, and an optional role. Actor attribution in audit logs becomes server-derived (not caller-asserted).

User registry format

EDITOR_USERS_JSON must be a JSON array. Set it as a single-line environment variable:

[
  { "id": "alice",  "displayName": "Alice Smith",  "key": "alice-unique-secret",  "role": "editor" },
  { "id": "bob",    "displayName": "Bob Jones",    "key": "bob-unique-secret",    "role": "admin"  },
  { "id": "carol",  "displayName": "Carol Wu",     "key": "carol-secret",         "role": "editor",
    "tokenIssuedAfter": "2026-03-25T12:00:00Z" }
]

Field	Required	Description
id	Yes	URL-safe unique identifier. Used as the actor value in tokens and audit records.
displayName	Yes	Human-readable name shown in the editor toolbar identity pill.
key	Yes	Personal login credential (like a personal API key). Must be unique; must not match API_KEY_ADMIN, API_KEY_EDITOR, or other API keys.
role	Yes	"editor" or "admin". Editor scope: template/asset management and preview. Admin scope: also restore and delete operations.
tokenIssuedAfter	No	ISO-8601 UTC timestamp. Rejects tokens with iat < tokenIssuedAfter for this user only — invalidates that user’s sessions without affecting others.

Startup validation: The API exits immediately if EDITOR_USERS_JSON is set but malformed, contains duplicate id or key values, contains invalid role values, or if any user key collides with an active API key (API_KEY_ADMIN, API_KEY_EDITOR, API_KEY_RENDER, API_KEY_PREVIEW).

Mode behaviour

	Named-user mode (EDITOR_USERS_JSON set)	Shared-key mode (default)
Login credential	Personal user key	API_KEY_EDITOR (or API_KEY_ADMIN)
Shared key accepted for login	No — 401 “Use your personal user key to log in”	Yes
Actor identity	Server-derived from user registry (user.id)	Caller-supplied at login (optional)
Editor role	user.role (editor or admin)	editor for API_KEY_EDITOR; admin for API_KEY_ADMIN
displayName in editor toolbar	Yes — from user registry	No
Programmatic X-Api-Key	Unaffected	Unaffected

Migrating to named-user mode (recommended cutover)

1. Generate a personal key for each team member:

   openssl rand -hex 32   # repeat for each user

2. Build the user registry in EDITOR_USERS_JSON. Assign role: "admin" to users who need restore/delete access; everyone else gets role: "editor".

3. Force all outstanding sessions to expire by setting EDITOR_TOKEN_ISSUED_AFTER to the current UTC time (prevents shared-key sessions from continuing after the cutover):

   EDITOR_TOKEN_ISSUED_AFTER=2026-03-24T12:00:00Z

4. Restart the API. The startup log confirms identityMode: named-users and the count of registered users. Any configuration errors cause an immediate exit with a descriptive message.

5. Distribute personal keys out-of-band (do not send via the same channel as login instructions). Editors re-login with their personal key.

6. Run the database migration to add createdBy audit columns to template_versions and assets:

   Postgres:

   pnpm --filter @pulp-engine/api db:deploy
   # applies: add_created_by (additive nullable columns, safe for existing data)

   SQL Server:

   pnpm --filter @pulp-engine/api db:migrate:sqlserver
   # applies: 002_add_created_by (additive nullable columns, safe for existing data)

   Run before starting the API. The migration runner applies all pending migrations automatically.

   File mode: no migration needed.

7. Remove API_KEY_EDITOR from the environment if only the editor uses it (all editor logins now go through the user registry). Keep it if any non-editor integrations depend on it for direct X-Api-Key access.

Per-user revocation

Two mechanisms are available. Both require an API restart to take effect.

Mechanism	Effect	How
Remove user from registry + restart	Blocks new logins and immediately invalidates active sessions for that user	Remove entry, restart
Set tokenIssuedAfter on user + restart	Invalidates sessions with iat < tokenIssuedAfter for that user only; newer sessions and other users are unaffected	Add "tokenIssuedAfter": "<iso-ts>", restart
Global EDITOR_TOKEN_ISSUED_AFTER	Invalidates all editor sessions (all users)	Existing mechanism, unchanged

Example — revoking a single user:

{ "id": "alice", "displayName": "Alice Smith", "key": "alice-unique-secret", "role": "editor",
  "tokenIssuedAfter": "2026-03-24T14:00:00Z" }

Tokens issued before 2026-03-24T14:00:00Z for alice are rejected. Tokens issued after that timestamp (new logins) are accepted. Other users are unaffected.

Rolling back to shared-key mode

Remove EDITOR_USERS_JSON from the environment and restart. The editor immediately returns to shared-key mode — no data changes are needed.

---

9. Known Production Risks

Risk	Likelihood	Mitigation
Puppeteer cold start (~2–3 s)	Low — singleton browser is reused after first request Resolved	The browser is now warmed at server startup: warmBrowser() (in-process) or the child-process dispatcher’s warmup() IPC pre-launches Chromium before the server accepts requests. The ~2–3 s cost moves to boot time; first PDF request is fast. /health/ready reflects warmup success. Container/socket modes are ephemeral and unaffected.
Render-time memory	Low for current template sizes	Both PDF routes stream via createPDFStream() — Node.js PDF buffer eliminated. Chrome render-time and HTML render memory still apply; monitor RSS on large documents.
Credential leaked or rotated	Low — shared secrets via env vars	Rotate the relevant scoped key via env var restart. For near-zero-downtime rotation of API_KEY_EDITOR or API_KEY_ADMIN (preserving existing editor sessions): set the old key as API_KEY_EDITOR_PREVIOUS / API_KEY_ADMIN_PREVIOUS and the new key as the active key, then restart — see runbook.md § Auth secret rotation. To invalidate only editor sessions without rotating API_KEY_EDITOR, set EDITOR_TOKEN_ISSUED_AFTER to the current UTC time and restart.
Editor session token compromised	Low — tokens are short-lived	Reduce EDITOR_TOKEN_TTL_MINUTES for sensitive environments. Set EDITOR_TOKEN_ISSUED_AFTER to invalidate all outstanding sessions without key rotation (requires restart).
Schema drift between dev and CI (postgres mode only)	Low — schema is stable	migrate deploy (CI/prod) and migrate dev (local) both apply the same committed SQL; keep prisma/migrations/ committed after any migrate dev run
Chromium missing system libraries	Medium on minimal Linux images	Test Puppeteer launch on the target OS before deploying; run node -e "require('puppeteer').launch()" to verify
Asset files not backed up	Low but high impact	Back up ASSETS_DIR alongside the assets table; loss of files causes broken image references in rendered PDFs
Multi-instance asset divergence (DB modes)	Medium — easy to overlook when scaling out	Running multiple API instances with ASSET_BINARY_STORE=filesystem (default) and no shared volume means each instance stores binaries locally. Assets uploaded by one instance return 404 when served by another. Mitigation: either mount a shared NFS/network volume as ASSETS_DIR on all instances, or switch to ASSET_BINARY_STORE=s3 — S3 mode eliminates the shared-volume requirement entirely. See § Object Storage.
Preview routes accessible in production	None — routes return 404 by default	PREVIEW_ROUTES_ENABLED absent is the safe default. If set to true, preview routes are active; pair with network-layer restrictions and confirm posture at deployment.
Multi-instance rate-limit multiplication	Medium — easy to overlook when scaling out	Rate limiting defaults to in-memory counters per API process. In multi-instance deployments, set RATE_LIMIT_STORE=redis with a shared REDIS_URL for consistent per-IP enforcement across the cluster. For public-facing deployments, edge rate limiting at the reverse proxy layer (nginx limit_req, HAProxy, Traefik, API gateway) remains the recommended primary throttle — Redis provides consistent internal enforcement, not a replacement for edge protection.

Production security checklist (v0.25.0)

Seven security controls are enforced by default when NODE_ENV=production (plus two conditional OIDC controls — OIDC_REDIRECT_URI https and an explicit OIDC_EDITOR_GROUPS — when OIDC is enabled). Set HARDEN_PRODUCTION=false to disable enforcement for evaluation.

Surface	Default	Risk	Recommended action
CORS	Allow all origins	Browser-originated cross-site requests accepted from any origin	Set CORS_ALLOWED_ORIGINS to a comma-separated list of trusted origins (e.g. https://editor.example.com)
Swagger UI (/docs*)	Public	Exposes full API surface and schema to anonymous callers	Set DOCS_ENABLED=false if the interactive docs are not needed in production
Prometheus metrics (/metrics)	Open, no auth	Exposes request rates, timing, and error counters to any caller	Set METRICS_TOKEN to require bearer token authentication, and/or restrict at the network layer
Editor login (POST /auth/editor-token)	Accepts HTTP	Raw API_KEY_EDITOR transmitted in plaintext over non-TLS connections	Set TRUST_PROXY=true and REQUIRE_HTTPS=true when the API is behind a TLS-terminating reverse proxy
Remote resource fetching	Allowed (public hosts)	Render pipeline can fetch arbitrary external resources during PDF generation	Set BLOCK_REMOTE_RESOURCES=true; optionally configure ALLOWED_REMOTE_ORIGINS for trusted font/image CDNs
Editor identity	Shared-key (anonymous)	No per-user audit trail for template changes	Set EDITOR_USERS_JSON for named-user identity (recommended), or ALLOW_SHARED_KEY_EDITOR=true to explicitly acknowledge shared-key mode

Startup warnings: When HARDEN_PRODUCTION=false is explicitly set in NODE_ENV=production, the API emits a consolidated [PulpEngine] warning at startup listing every unconfigured control. These do not block startup — they inform operators that enforcement is disabled. Example:

[PulpEngine] HARDEN_PRODUCTION=false — production security enforcement is disabled.
The following controls are not configured:
   • CORS_ALLOWED_ORIGINS must be set to a comma-separated list of specific trusted origins ...
   • METRICS_TOKEN must be set to protect GET /metrics with bearer authentication ...
   Remove HARDEN_PRODUCTION=false to restore default production enforcement.

Hardened Production Mode

On by default in production. When NODE_ENV=production (the Docker default), hardening is enforced automatically — the server fails to start unless all seven security controls are configured. When OIDC/SSO is enabled, two conditional controls are also enforced: OIDC_REDIRECT_URI must be https:// and OIDC_EDITOR_GROUPS must be set explicitly. Set HARDEN_PRODUCTION=false to explicitly opt out for evaluation. Accepted values: true, false, 1, 0, or unset (auto-derived from NODE_ENV).

Enforced rules:

Control	Rule
CORS	CORS_ALLOWED_ORIGINS must be set to specific origins — wildcard * is rejected (it is equivalent to leaving CORS open)
Docs	DOCS_ENABLED must be explicitly set in the environment (false to disable, true to acknowledge exposure)
Metrics	METRICS_TOKEN must be configured
HTTPS	REQUIRE_HTTPS=true must be set
Proxy	TRUST_PROXY=true must be set when REQUIRE_HTTPS=true
Remote resources	BLOCK_REMOTE_RESOURCES=true must be set to prevent the render pipeline from fetching resources from arbitrary public hosts during PDF generation. Optionally set ALLOWED_REMOTE_ORIGINS for trusted font/image CDNs.
Named users	When editor login is capable (API_KEY_EDITOR, API_KEY_ADMIN, or API_KEY set), either EDITOR_USERS_JSON must be configured for per-user identity (recommended), or ALLOW_SHARED_KEY_EDITOR=true must be set to explicitly accept shared-key identity.

All violations are collected first and reported together, so operators can fix everything in a single pass. Example failure output:

❌ HARDEN_PRODUCTION=true but required security controls are not configured:
   • CORS_ALLOWED_ORIGINS must be set to a comma-separated list of specific trusted origins ...
   • DOCS_ENABLED must be explicitly set. Use DOCS_ENABLED=false to disable the Swagger UI ...
Configure all required controls or unset HARDEN_PRODUCTION to disable enforcement.

Example hardened .env configuration:

# Security hardening — required when HARDEN_PRODUCTION=true
CORS_ALLOWED_ORIGINS=https://editor.example.com
DOCS_ENABLED=false
METRICS_TOKEN=<output of: openssl rand -hex 32>
REQUIRE_HTTPS=true
TRUST_PROXY=true
BLOCK_REMOTE_RESOURCES=true
# Named-user identity: configure EDITOR_USERS_JSON (recommended) or opt out:
# ALLOW_SHARED_KEY_EDITOR=true
HARDEN_PRODUCTION=true

Why wildcard CORS is rejected: CORS_ALLOWED_ORIGINS=* explicitly allows all browser origins — the same open posture as leaving it unset. Hardened mode requires specific, named origins so the trust boundary is enforced.

Why TRUST_PROXY is required with REQUIRE_HTTPS: Fastify derives request.protocol from the raw socket connection. Behind a TLS-terminating reverse proxy, the socket is plain HTTP; without TRUST_PROXY=true, Fastify cannot read X-Forwarded-Proto and will reject all editor-token logins regardless of the external connection. Setting TRUST_PROXY=true is also safe for direct-TLS deployments (no proxy) — it only affects behavior when X-Forwarded-Proto headers are present.

Deployment validation: When METRICS_TOKEN is configured, pass it as the 4th argument to validate-deploy.sh:

./scripts/validate-deploy.sh https://api.example.com "$ADMIN_KEY" "" "$METRICS_TOKEN"

HTTP security headers (@fastify/helmet)

The API registers @fastify/helmet to set baseline HTTP security headers on every response. CSP is route-aware because different surfaces have different requirements:

Route prefix	Content-Security-Policy	Permissions-Policy
/editor, /editor/*	default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; frame-src blob:; object-src blob:; img-src 'self' blob: data:; connect-src 'self'	clipboard-write=(self)
/docs*	Deferred to Swagger UI’s staticCSP — not set by the API	(none)
/render/html, /render/preview/html	Not set — the HTML renderer embeds a <meta> CSP (script-src 'none'; object-src 'none') inside the document	camera=(), microphone=(), geolocation=()
All other routes (API JSON, health, metrics)	default-src 'none'	camera=(), microphone=(), geolocation=()

Additional headers set by Helmet:

Header	Value	Notes
Strict-Transport-Security	max-age=31536000; includeSubDomains	Only set when REQUIRE_HTTPS=true
X-Frame-Options	DENY	Prevents clickjacking for all routes
Referrer-Policy	strict-origin-when-cross-origin
X-Content-Type-Options	nosniff	Helmet default
X-DNS-Prefetch-Control	off	Helmet default

Cross-origin isolation policies (Cross-Origin-Resource-Policy, Cross-Origin-Embedder-Policy, Cross-Origin-Opener-Policy) are disabled. These headers are designed for SharedArrayBuffer isolation and would break cross-origin asset loading when the editor runs on a different port (development) or a separate domain.

Deployment model — trusted authors

Pulp Engine is designed for trusted internal teams: developers, operators, and named template authors operating within a single organisation. Templates are rendered by a headless Chromium instance; Handlebars templates and helpers run server-side and should be authored by credentialled, internal-trusted users — not anonymous public users.

Pulp Engine is not designed or marketed for hostile multi-tenant environments where arbitrary, untrusted users author templates. If your deployment model includes untrusted template authoring, apply additional sandboxing at the infrastructure level before exposing the editor to those users.

For production deployments, enabling HARDEN_PRODUCTION=true and configuring EDITOR_USERS_JSON (per-user named credentials) gives you enforced security controls and a full audit trail of which named user authored each template change.

---

10. Upgrading from file mode to database-backed storage

Upgrading the server version? Stored template definitions are format-compatible across upgrades (additive-only schema changes, enforced in CI by a frozen fixture corpus). See template-compatibility.md for the formatVersion semantics and how to pre-flight stored templates with pulp validate.

Use the db:migrate:file-to-db script to copy all template and asset metadata from a file-mode TEMPLATES_DIR into a postgres or sqlserver database. This is a one-time operator tool — it is not a sync engine and is not designed for ongoing replication.

Idempotency: skip, not merge

Records that already exist in the target database are skipped without modification. The script does not update or reconcile existing records with file-mode data. For a clean first migration, ensure the target database is empty. Running the script against a partially populated target inserts any missing records and skips any that already exist — by design.

Prerequisites

1. Stop the API. The migration reads from TEMPLATES_DIR while the source is at rest. Running the API concurrently against the file-mode directory may produce inconsistent results.
2. Apply the target schema first:
   • Postgres: pnpm --filter @pulp-engine/api db:deploy
   • SQL Server: pnpm --filter @pulp-engine/api db:migrate:sqlserver
3. Use an empty target database for a clean migration. See idempotency note above.
4. The API must have run at least once in file mode before migrating. The file-mode store auto-migrates older flat-layout template files ({key}.json at root) to the current folder layout ({key}/meta.json + {key}/versions/) on startup. If you skip this step, any flat-layout templates will not be visible to the migration script.
5. Asset binaries stay on disk. The script migrates metadata only (not binary files). Ensure ASSETS_DIR in the target deployment points to the same directory as the file-mode ASSETS_DIR, or manually copy all binary files to the new ASSETS_DIR before cutover. See asset binary note below.

Environment variables

Variable	Required	Value
STORAGE_MODE	Yes	postgres or sqlserver — the target backend
TEMPLATES_DIR	Yes	Path to the existing file-mode template directory (source)
ASSETS_DIR	Optional	Path to the existing file-mode assets directory. Asset binaries are not moved; only metadata from .assets-index.json is read.
DATABASE_URL	Yes (postgres)	Prisma connection string for the target database
SQL_SERVER_URL	Yes (sqlserver)	mssql connection URL for the target database

Env-var precedence note: Shell environment variables override values from .env (dotenv-cli default — no --override). Pass STORAGE_MODE and the connection string explicitly in the shell when running the migration. The script prints the effective configuration at startup — verify the output before migration proceeds. If .env still contains STORAGE_MODE=file and no shell override is provided, the guard will catch it and exit 1 with a clear error.

Steps

# 1. Stop the API

# 2. Dry run first — review what will be inserted and verify the startup lines
STORAGE_MODE=postgres \
  TEMPLATES_DIR=/var/pulp-engine/templates \
  ASSETS_DIR=/var/pulp-engine/assets \
  DATABASE_URL=postgresql://user:pass@host:5432/pulp-engine \
  pnpm --filter @pulp-engine/api db:migrate:file-to-db -- --dry-run
# Verify the startup log shows the correct storageMode, TEMPLATES_DIR, and ASSETS_DIR

# 3. Run the migration
STORAGE_MODE=postgres \
  TEMPLATES_DIR=/var/pulp-engine/templates \
  ASSETS_DIR=/var/pulp-engine/assets \
  DATABASE_URL=postgresql://user:pass@host:5432/pulp-engine \
  pnpm --filter @pulp-engine/api db:migrate:file-to-db

# 4. Review the summary — "Templates skipped" should be 0 on a clean first run
#    Watch for any warnings printed above the summary (exit code 2 = partial)

# 5. Update .env: set STORAGE_MODE=postgres (or sqlserver)
#    Remove TEMPLATES_DIR if it was only used for file mode

# 6. Start the API

Exit codes

Code	Meaning
0	All records inserted or idempotently skipped; no source-data warnings
1	Fatal error: bad configuration, DB connection failure, or malformed .assets-index.json
2	Partial success: some templates were skipped due to unreadable meta.json or version files; all other records were inserted

Source-data error policy

Problem	Behavior	Exit
Unreadable / malformed meta.json	Skip that template	2
All version files unreadable	Skip that template	2
currentVersion not in readable version set	Skip that template (avoids a broken DB record)	2
Some version files unreadable, currentVersion readable	Migrate readable versions only	2
Malformed .assets-index.json	Fatal abort — fix the file before re-running	1

Asset binaries are not moved

The script migrates metadata only. Asset binary files (images) are not copied or moved. They must be accessible at the ASSETS_DIR path configured for the target deployment.

• Same directory: If the target deployment will use the same filesystem path as the source ASSETS_DIR, no action is needed — binaries are already in place.

• Different directory or new server: Copy the entire contents of the source ASSETS_DIR to the new location before running the migration (and before starting the API), then verify the asset binary files are present. Missing binaries will cause broken image references in rendered PDFs after cutover.

Known limitations

1. Template and version row IDs are not preserved. Templates and template versions receive new CUIDs (postgres) or UUIDs (SQL Server) in the target database. These IDs are internal and never appear in API responses or URLs.
2. Asset IDs are preserved from the source .assets-index.json records.
3. Pre-folder-layout templates (from very early file-mode versions) must be auto-migrated by the file-mode store first (run the API once before migrating).
4. One direction only. There is no reverse migration path (DB → file mode).
5. SQL Server updated_at precision: stored as DATETIME2(3) (millisecond precision). Sub-millisecond precision in the source timestamp is truncated.

---

Quick-reference checklist

• Node 22–24 and pnpm 10.32.1 installed on the server
• Chromium system dependencies installed (Linux only)
• .env created from .env.example with STORAGE_MODE, HOST, PORT, NODE_ENV=production, API_KEY_ADMIN (and optionally API_KEY_RENDER / API_KEY_PREVIEW / API_KEY_EDITOR); ASSETS_DIR (absolute path), ASSETS_BASE_URL
• pnpm install completed

Postgres mode only (STORAGE_MODE=postgres or unset):

• PostgreSQL accessible from the API process
• DATABASE_URL set in .env
• pnpm db:generate completed
• pnpm db:deploy applied migrations to the production database
• pnpm db:seed loaded sample templates
• Daily full-database pg_dump scheduled (no --table filters — see § 7 Backup and Recovery and the backup & restore runbook)
• If running multiple API instances with ASSET_BINARY_STORE=filesystem (default): ASSETS_DIR is on a shared volume accessible to all instances (not required when ASSET_BINARY_STORE=s3)

SQL Server mode only (STORAGE_MODE=sqlserver):

• SQL Server (2019+) or Azure SQL accessible from the API process
• SQL_SERVER_URL set in .env
• pnpm db:generate completed (compiles Prisma types; no DB connection made)
• pnpm --filter @pulp-engine/api db:migrate:sqlserver applied schema
• pnpm db:seed loaded sample templates
• Database backup scheduled
• If running multiple API instances with ASSET_BINARY_STORE=filesystem (default): ASSETS_DIR is on a shared volume accessible to all instances (not required when ASSET_BINARY_STORE=s3)

File mode only (STORAGE_MODE=file):

• TEMPLATES_DIR set in .env and directory contains at least one template JSON file
• pnpm db:generate completed (compiles Prisma types; no DB connection made)
• Both TEMPLATES_DIR and ASSETS_DIR included in your backup schedule (asset metadata + binaries live in ASSETS_DIR — see the runbook § 6)

All modes:

• pnpm build succeeded
• node apps/api/dist/index.js starts without errors
• If deploying in a container without sandbox support: PULP_ENGINE_DISABLE_SANDBOX=true set in .env
• GET /health returns 200 { "status": "ok", "version": "..." }
• GET /health/ready returns 200 { "status": "ok", "checks": { "storage": "ok", "assetBinaryStore": "ok", "renderer": "ok" } }
• Licence verified (licensed deployments): PULP_LICENCE_KEY set, the startup log does not say Running under Evaluation Licence — output will be watermarked, and the /health/ready licence block reports the licensed state — see licence-key-format.md
• GET /metrics returns Prometheus text format (contains process_cpu_seconds_total)
• API bound to 127.0.0.1 (or private IP), not exposed directly
• Reverse proxy configured and restricting access to internal callers
• /metrics endpoint restricted to scraper IPs at the network/proxy layer
• Process manager (PM2 / systemd) configured for auto-restart
• Preview route posture confirmed: PREVIEW_ROUTES_ENABLED absent (routes return 404 in production) or intentionally set to true with network restrictions in place

Deterministic date defaults

Set TZ=UTC on API and render-worker processes. Templates and request options can override the rendering time zone. Node’s bundled ICU supplies locale and zone rules; update Node patch versions when those rules change. Calendar dates and floating ISO date-times retain their fields independently of host time zone.

Font assets and installed fonts

Setting	Default	Meaning
FONT_INLINE_MAX_BYTES	6291456	Aggregate raw font bytes per render, 1 byte to 12 MiB; images and base64 fonts also share the isolation protocol’s 16 MiB request frame.
PULP_FONT_PACK	host	Image-provided minimal, standard or full pack identifier, advertised by /capabilities; unset on bare-metal hosts.

API and worker Docker images default to FONT_PACK=full. Rebuild both with the same --build-arg FONT_PACK=minimal or standard to reduce font coverage and size. minimal includes Liberation and fontconfig; standard adds Noto core and colour emoji; full adds Noto CJK. Fontconfig caches are built before switching to the unprivileged runtime user and verified on read-only filesystems. The API uses XDG_CACHE_HOME=/tmp/.font-cache; workers use /tmp/.chromium-cache. Mount /tmp as writable tmpfs when using --read-only. The controller image needs no fonts.

Bare-metal and Windows installations must install equivalent script fonts themselves. The host capability intentionally does not inventory local fonts. Uploaded template fonts are embedded independently of installed font packs and need no outbound network access. See the runbook’s font diagnostics.

DOCUMENT_CREATOR defaults to Pulp Engine and supplies generated document creator metadata when the template does not override it.

Rendered document archive

Apply PostgreSQL migrations (including 20261011000000_add_rendered_documents) or SQL Server migration 013_rendered_documents.sql before enabling the archive. Metadata storage must be PostgreSQL or SQL Server; file/plugin modes do not provide document storage.

Enable DOCUMENT_STORE_ENABLED=true and supply DOCUMENT_URL_SECRET (at least 32 hexadecimal characters; generate a random 32-byte secret). Keep this secret identical across API instances. To rotate it, place the old value in DOCUMENT_URL_SECRET_PREVIOUS, deploy the new secret to all instances, and remove the previous secret after outstanding links have expired. Link revocation is persisted per document. No schema or template-model migration is required for templates.

Setting	Default / purpose
DOCUMENT_BLOB_STORE	filesystem or s3; default filesystem.
DOCUMENT_BLOB_DIR	./document-blobs; private, writable filesystem directory.
DOCUMENT_BLOB_BUCKET	Required existing private bucket for S3.
DOCUMENT_DEFAULT_TTL_HOURS / DOCUMENT_MAX_TTL_HOURS	168 / 8760; requested and tenant TTLs are capped by the operator maximum.
DOCUMENT_RETENTION_JSON / DOCUMENT_RETENTION_JSON_FILE	Mutually exclusive per-tenant overrides, e.g. {"acme":{"defaultTtlHours":720,"maxTtlHours":2555}}.
DOCUMENT_LINK_DEFAULT_TTL_SECONDS / DOCUMENT_LINK_MAX_TTL_SECONDS	3600 / 604800; also capped by document expiry.
DOCUMENT_MAX_SIZE_BYTES	104857600; checked against final bytes before writing.
DOCUMENT_PURGE_INTERVAL_MINUTES	60; expired rows/blobs, exhausted caps, and orphan cleanup.
DOCUMENT_S3_REDIRECT	false; enable short-lived direct S3 downloads for non-HTML formats.
SCHEDULE_DOCUMENT_TTL_HOURS	168; also capped by the document retention policy.
PUBLIC_BASE_URL	Externally reachable API origin/base URL; required for link-mode schedule emails.

S3 uses existing S3_REGION, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, optional S3_ENDPOINT, and S3_PATH_STYLE. Grant only the private bucket’s object get/put/delete/list and bucket health-check permissions. Block public access and enable server-side encryption and regional residency policies at the storage provider. Store raw document bytes; the job-result blob store remains separate.

Filesystem archives need a persistent shared volume when several instances serve the same metadata database. S3 is the preferred shared backend for high availability. Readiness checks metadata and blob availability when configured. Ensure reverse proxies preserve the public scheme/host; only configure TRUST_PROXY for trusted proxies. Prefer explicit PUBLIC_BASE_URL for stable links.

Purge deletes blobs before rows and retries failed deletion on later passes. Orphan blobs must be older than one hour before removal. Exhausted download caps have a 60-second purge grace to allow an issued S3 redirect to complete. Expiry is enforced immediately by the API, even before the next purge. Purge may run on multiple instances. Provider outages and backups can retain physical bytes beyond the logical expiry; monitor storage and the document operation counter/audit events. Rendering with storage buffers the complete output, so size the render concurrency and memory budget accordingly.

Large render inputs and datasets

Variable	Default	Purpose
RENDER_BODY_LIMIT_BYTES	10485760	Single/legacy render JSON limit; 64 KiB to 256 MiB.
BATCH_BODY_LIMIT_BYTES	52428800	Sync batch, legacy async and job JSON limit; 64 KiB to 512 MiB.
REQUEST_DECOMPRESSION_ENABLED	true	Accept gzip, x-gzip, deflate and br on JSON/NDJSON.
DATA_URL_ENABLED	false	Opt in to per-render URL data sources.
DATA_URL_ALLOWED_ORIGINS	empty	Exact comma-separated HTTP(S) origins. When present, only these origins are allowed, including private addresses vouched for by the operator. No paths, credentials or wildcards.
DATA_URL_MAX_BYTES	batch limit	Maximum decoded URL response bytes.
DATA_URL_TIMEOUT_MS	15000	Total URL fetch deadline, including DNS and body reads.
DATASET_BODY_LIMIT_BYTES	104857600	Decoded temporary upload limit; 64 KiB to 512 MiB.
DATASET_MAX_TTL_SECONDS	86400	Maximum upload lifetime; 1 to 86400 seconds. Default requested lifetime is 3600 seconds, bounded by this maximum.

Dataset blobs reuse the private job-blob backend settings with a separate datasets/ S3 prefix or filesystem directory. Share that backend across API instances when using dataRef; no database migration is required. Metadata and JSON are published separately, with expiry enforced on every read and a minute sweep. Private S3 origins must be operator-controlled; allowlisting permits access to internal addresses. Render URL data sources require HTTPS in production, reject redirects, and pin DNS answers. Existing scheduled URL sources retain their enabled posture but now use the same bounded response reader.

GET /capabilities reports effective limits and request encodings. Socket/container PDF workers retain a 16 MiB serialized input-frame ceiling; increasing HTTP limits does not increase that ceiling. Unsupported encodings return 415, oversized decoded requests return actionable 413 errors, and corrupt compression returns 400.

Draft/live lifecycle configuration

Variable	Default	Purpose
RENDER_DEFAULT_LABEL	unset	Default label for unpinned root production renders, jobs and schedules. Nested components/layouts follow HEAD unless explicitly pinned.
RENDER_DEFAULT_LABEL_MODE	fallback	Use HEAD when the label is missing, or strict to fail with 409.
RENDER_DEFAULT_LABEL_SEED_ON_CREATE	true	Point the default label at the first saved version.
CHANGE_NOTE_REQUIRED	false	Require notes on template updates and restores.
SERVER_DRAFTS_ENABLED	true	Enable identity-owned drafts that do not advance HEAD.

Audit and align existing labels before enabling the default: stale labels can roll production output back. Start with fallback, monitor pulp_engine_default_label_fallback_total, then enable strict. Explicit versions and labels take precedence; label:"@latest" opts into HEAD. Preview and validation continue using HEAD.

Event webhooks and render log

Variable	Default	Purpose
EVENT_WEBHOOKS_ENABLED	true	Enable database event subscriptions/dispatcher.
EVENT_WEBHOOK_POLL_INTERVAL_MS	2000	Jittered polling cadence (100–60000 ms).
EVENT_WEBHOOK_MAX_CONCURRENT	8	Maximum in-flight attempts per API process (1–64).
EVENT_WEBHOOK_MAX_ATTEMPTS	8	Durable attempt cap (1–8).
EVENT_WEBHOOK_DELIVERY_RETENTION_DAYS	30	Terminal delivery payload retention.
EVENT_WEBHOOK_MAX_SUBSCRIPTIONS_PER_TENANT	20	Atomic per-tenant subscription cap.
EVENT_WEBHOOK_MAX_PENDING_PER_SUBSCRIPTION	10000	Pending/delivering cap; excess events drop with a metric.
RENDER_LOG_ERROR_MESSAGES	true	Store truncated failure messages; false also omits render.failed messages.
RENDER_LOG_PREVIEW_FAILURES	false	Opt in to diagnostic preview failure rows.

Requires current database migrations. File mode has neither durable event webhooks nor render logs. See webhooks and retention.

Stored secrets and mounted credentials

Stored-secret encryption is opt-in in this release. Hardened deployments warn when persisted credentials are enabled without encryption. Configure it before creating credentials, and backfill existing records. AES-256-GCM protects schedule URL header values, webhook signing secrets (including event webhook rollover secrets), scheduled PDF owner/user passwords, batch callback secrets and editor login keys. Database editor keys use a separate HMAC blind index; plaintext stays in process memory for authentication and signing. Render data, URLs (including query tokens/userinfo), logs and arbitrary plugin data are outside this field-level scope. Use encrypted disks/database backups for complete data-at-rest coverage.

Variable	Default	Purpose
PULP_ENCRYPTION_PROVIDER	local	local or aws-kms.
PULP_ENCRYPTION_KEY	unset	Canonical base64 of exactly 32 random bytes. Never commit it.
PULP_ENCRYPTION_KEY_ID	k1	Public envelope ID, 1-16 lowercase letters/digits/hyphens.
PULP_ENCRYPTION_KEYS_PREVIOUS	unset	At most 3 decrypt-only kid:base64 entries, comma separated.
PULP_ENCRYPTION_KMS_KEY_ID	unset	AWS KMS symmetric key ARN/ID; requires postgres or sqlserver. Region comes from AWS_REGION.

KMS stores wrapped AES data keys in data_encryption_keys, unwraps them once at boot and encrypts values locally. The AWS default credential chain is supported. Grant kms:Decrypt and kms:GenerateDataKey on the selected key. KMS context binds the application, purpose and public data-key ID. Vault Transit, Azure Key Vault and GCP KMS are deferred. An unavailable KMS provider fails initialization; a stored value encrypted under an absent local key is masked in lists and fails credential use with secret_unavailable. The decrypt failure metric is pulp_engine_secret_decrypt_failures_total{store}. Never treat the public unavailable marker as a credential.

Secrets from files

For every setting in FILE_BACKED_SECRET_VARS, use <NAME>_FILE=/run/secrets/name instead of <NAME>. This includes database/Redis URLs, API and rollover keys, editor JSON, metrics/document link secrets, S3/SMTP credentials, OIDC secrets, Anthropic/sandbox/commerce/licence secrets, encryption keys, audit sink secrets/headers and AWS default-chain credentials. Exactly one trailing LF or CRLF is stripped. Empty/unreadable files and simultaneous inline/file values fail boot. Resolved values are placed in the process environment for Prisma, AWS and renderer workers. Do not print the environment.

For Docker, bind a read-only file and set API_KEY_ADMIN_FILE=/run/secrets/admin; for Compose use a secrets mount with that target. For Kubernetes, mount a Secret volume at /run/secrets and set DATABASE_URL_FILE=/run/secrets/database-url and PULP_ENCRYPTION_KEY_FILE=/run/secrets/encryption-key. All replicas must mount the same keyring. Mounted file changes take effect on restart. API_KEYS_JSON_FILE, API_JWT_ISSUERS_JSON_FILE, RATE_LIMITS_JSON_FILE and EDITOR_USERS_FILE retain their existing readers; no double _FILE suffix is added. Public endpoints and KMS identifiers are not secrets and have no file resolver.

Backfill, rotation and downgrade

Run pnpm --dir apps/api secrets:reencrypt --dry-run with the same storage configuration/keyring as the API, then run without --dry-run. Counts contain no secret values. Database rewrites use optimistic comparisons and can run while the API is live; rerun if concurrent writes were skipped. File registry maintenance requires stopping the API first because file mode has one writer. Plugin-owned secrets need the plugin’s migration tools; this command only maintains the core file registry in plugin mode.

Pause editor-user creation during a rolling keyring transition: replicas with an older keyring cannot compare blind indexes from a future key. Resume creation after every replica has the same keyring. For local rotation, deploy a new key/ID with the old key in PULP_ENCRYPTION_KEYS_PREVIOUS on every replica, restart all replicas, backfill, verify zero failed/skipped rows and then remove the old key. Never remove a key while rows still reference it. For KMS rotation, run secrets:reencrypt --rotate-dek, restart every replica to load the new active data key, then run maintenance again to catch writes from replicas that had the old key cached. Keep previous wrapped data keys and the KMS keys needed to unwrap them. A first-time KMS dry run requires an initialized keyring and never creates a data-key row.

Before disabling encryption or downgrading, run secrets:reencrypt --decrypt using the complete keyring and verify no failures/skips. Then restart without encryption. Older versions cannot read encrypted credentials or editor rows with null plaintext keys. Back up encryption material separately from database/file backups; loss is unrecoverable. Re-enter webhook/header/PDF passwords and reissue editor keys if the keyring is lost. Built-in stores bind envelopes to tenant, row and purpose. Legacy storage plugins may choose their own schedule IDs; their schedule decorator binds tenant and purpose, so plugins should supply their own row binding if required.

Audit export and SIEM forwarding

GET /audit-events/export?format=csv|ndjson streams filtered, ascending audit rows. Admin credentials are required. Omitted format uses Accept: application/x-ndjson or defaults to CSV. The default time window is the last 30 days. AUDIT_EXPORT_MAX_ROWS defaults to 500000 (maximum 5000000); narrow filters if the server returns 413 audit_export_too_large. CSV cells with spreadsheet formula prefixes are escaped. Every successful export records audit_export. The feed exposed over HTTP omits tenant IDs. File mode bounds page memory but rescans JSONL for each page; use a database for large audit histories.

Variable	Default	Purpose
AUDIT_SINK_TYPE	off	off, http or one structured log line per event with log.
AUDIT_SINK_URL	unset	Operator-owned collector. HTTPS required; private addresses are allowed. No redirects, userinfo or fragments.
AUDIT_SINK_ALLOW_HTTP	false	Explicitly allow HTTP for a local collector/test network.
AUDIT_SINK_SECRET	unset	HMAC-SHA256 signing key; supports _FILE.
AUDIT_SINK_HEADERS_JSON	unset	String header map, e.g. Splunk Authorization; supports _FILE. Routing/framing and signature headers are reserved.
AUDIT_SINK_FORMAT	json	JSON array, ndjson, or splunk-hec event envelopes.
AUDIT_SINK_BATCH_SIZE	500	1-10000 events per batch.
AUDIT_SINK_INTERVAL_MS	5000	Poll interval, 100-300000 ms. Failures back off to 5 minutes.
AUDIT_SINK_LAG_MS	5000	Commit safety window, 0-3600000 ms. Set longer than your longest audit-writing transaction.

The forwarder tails stored rows, including template/asset transaction inserts, across all tenants. Outbound payloads deliberately include tenantId. A database lease chooses one replica; a durable cursor advances only after a 2xx response (or synchronous log submission). Signatures match webhooks: X-PulpEngine-Signature: sha256=<hex>, X-PulpEngine-Timestamp: <seconds>, HMAC input <seconds>.<exact body>. Requests time out after 10 seconds. Collectors should deduplicate by event id: retries/crashes after acceptance but before cursor persistence can repeat a batch. The log sink confirms submission to the logger, not delivery to an external collector.

The cursor identity is derived from sink type and URL; changing destination replays retained history. Changing headers, secret or format preserves the cursor. File mode uses .audit-sink-cursor.json under TEMPLATES_DIR and a short exclusive .lock; after a process crash during a file update, stop the writer and remove a leftover lock before restarting. Database mode is recommended for multiple replicas. Monitor pulp_engine_audit_sink_events_total{status} and pulp_engine_audit_sink_lag_seconds. Retention overtaking the cursor emits a warning; an outage longer than audit retention can lose events. Transactions open longer than the lag window can commit behind the cursor and be skipped. Syslog/OTLP adapters and a tamper-evident hash chain are deferred. For a PostgreSQL migration job with mounted secrets, run the compiled resolver wrapper: node /app/dist/scripts/prisma.js migrate deploy --schema=/app/src/prisma/schema.prisma with DATABASE_URL_FILE and its read-only mount. In a source checkout, pnpm --dir apps/api db:deploy uses the same resolver. The SQL Server migration runner resolves SQL_SERVER_URL_FILE before connecting.

Admin console deployment

The bundled editor includes /editor/admin/; /editor/admin redirects permanently to the trailing-slash URL. It uses the existing editor CSP and same-origin API connection policy. The Docker editor distribution includes the admin entry automatically.

Configuration	Available console access
Named users / OIDC, admin role	Tenant admin via the existing editor session
Shared-key mode	Admin API-key console sign-in; shared editor sessions remain editor-scoped
File storage	Users in named mode, audit and system; database-only surfaces are hidden
Postgres / SQL Server	Available delivery, usage and API-key stores are surfaced
Multi-tenant super-admin key	Tenant CRUD and a tenant selector for scoped data

CONSOLE_SESSION_TTL_MINUTES defaults to 60, accepts 5-1440 minutes, and is capped by EDITOR_TOKEN_TTL_MINUTES. Console sessions have no refresh. CONSOLE_SESSION_SECRET is an optional server-held secret of at least 32 characters. Static-key console sessions are signed by their own admin credential and verified only against that credential. Database-key sessions use CONSOLE_SESSION_SECRET, falling back only to the server-held OIDC_COOKIE_SECRET. Database API-key console sign-in requires CONSOLE_SESSION_SECRET when the OIDC fallback is absent. All replicas must share stable signing material. Rotating it invalidates outstanding console sessions. _FILE is supported, including CONSOLE_SESSION_SECRET_FILE; the existing mounted-secret precedence and permission checks apply.

Set REQUIRE_HTTPS=true and correctly configure TRUST_PROXY behind TLS termination. This rejects plain HTTP credential submission to both token-mint endpoints and all plain HTTP requests carrying X-Console-Token. Static console sessions are bearer sessions; they are not pinned to the initial IP so corporate proxy changes remain usable. Database-key CIDR restrictions are enforced at mint and on every authenticated console request (subject to registry cache freshness). Browser storage is tab-scoped sessionStorage; credentials are cleared on sign-out and tokens are never embedded in URLs. Responses to console-authenticated requests and console session minting use Cache-Control: no-store.

Fine-grained access control

See Access control for roles, grants and session migration.

Variable	Default	Meaning
RBAC_CACHE_TTL_MS	10000	Per-tenant role/binding cache lifetime, 0-300000 ms; other replicas converge within this window.
RBAC_OWNER_ROLE	author	Implicit template ownership role: author, publisher, viewer or none.
OIDC_ROLE_SYNC	true	Recompute the stored base role on every OIDC login. Hardened OIDC deployments warn when false.
OIDC_ROLE_MAPPINGS_JSON	unset	JSON array of group/role/optional-folder mappings.
OIDC_ROLE_MAPPINGS_JSON_FILE	unset	UTF-8 file containing those mappings; mutually exclusive with the inline value.

Report data-source fan-out (Wave 37)

Variable	Default	Bounds	Purpose
DATA_SOURCE_MAX_CONCURRENCY	5	1-100	Simultaneous provider requests within each render
DATA_SOURCE_MAX_FETCHES	200	1-10000	Distinct requests per render, shared by parameter choices and sub-reports

A production render exceeding the fetch cap returns 422 data_source_error. Preview stops fetching and falls back to parent data. Sessions/cache are isolated per render; identical URL requests deduplicate inside that session. A batch can multiply these limits by its render concurrency. Size upstream capacity and the existing render concurrency together. This wave reuses the schedule parameters and execution run-context JSON columns installed in Wave 9; no new migration is required. Existing SSRF controls continue to apply to option and sub-report URLs.

Shared datasets and template tests

Variable	Default	Purpose
DATASETS_MAX_PER_TEMPLATE	25	Maximum shared datasets per template (1-100).
TEMPLATE_TESTS_MAX_DATASETS	10	Maximum datasets rendered in one template test run (1-50).

Editor presence and lock settings

Variable	Default	Meaning
TEMPLATE_PRESENCE_ENABLED	true	Register template collaboration routes.
TEMPLATE_PRESENCE_TTL_SECONDS	90	Presence expiry; minimum 30 seconds.
TEMPLATE_LOCK_IDLE_MINUTES	5	Holder heartbeat renewal window.
TEMPLATE_LOCK_MAX_HOURS	8	Maximum lifetime from acquisition.

Locks remain advisory. Multi-replica deployments need RATE_LIMIT_STORE=redis and REDIS_URL for shared presence and atomic locks; memory mode is per replica.

Image and email output settings

Variable	Default	Description
IMAGE_ENABLED	true	Enable image render and thumbnail routes and advertise formats.image.
IMAGE_MAX_DPI	300	Maximum requested DPI; width overrides DPI and remains pixel-budgeted.
IMAGE_MAX_PAGES	20	Maximum selected raster pages.
IMAGE_MAX_TOTAL_MEGAPIXELS	150	Aggregate selected-page pixel budget checked before allocation.
THUMBNAIL_STORE	fs	Private cache backend: fs, s3 or none.
THUMBNAIL_DIR	./data/thumbnails	Private filesystem thumbnail directory.
THUMBNAIL_BUCKET	unset	Private S3 thumbnail bucket, using existing S3 connection settings.
EMAIL_ENABLED	true	Enable email render routes and advertise formats.email.
EMAIL_ASSET_BASE_URL	unset	Server-controlled public asset origin required by email absolute-image mode.

Keep thumbnail caches private. In HA, use S3 or shared filesystem storage for cache reuse; independent local caches are valid. Cache failures fall back to rendering. Native raster dependencies must be installed for each image architecture. See output target behavior and verification.

Tenant erasure and caller quotas

Variable	Default	Description
TENANT_PURGE_GRACE_HOURS	24	Minimum archived age before irreversible purge, 0-8760 hours.
RATE_LIMIT_KEY_MODE	mode-dependent	ip, tenant or credential; unset preserves current behavior.
RATE_LIMIT_HEADER_STYLE	x	x or draft rate-limit response headers.
QUOTAS_JSON	unset	Validated array of tenant/credential day/month render/page rules.
QUOTAS_JSON_FILE	unset	UTF-8 rules file; mutually exclusive with QUOTAS_JSON.

Use RATE_LIMIT_STORE=redis and REDIS_URL for strict multi-replica enforcement. Periods default to UTC; an explicit IANA timezone uses calendar boundaries including DST. Production and test calls count, preview defaults off and schedules default on for tenant rules. Database key restrictions may override ratePerMinute and add quotas. Rules require restart. See API quota semantics.

Lifecycle requires multi-tenant PostgreSQL or SQL Server. Apply the lifecycle migration before starting the upgraded API. Database guards reject late writes to purging/purged tenants; binary stores check before and after writes. Use shared private S3 stores for replicas. Current objects are purged; backups and noncurrent bucket versions follow operator retention policies.

Scoped browser render tokens

Variable	Default	Description
RENDER_TOKENS_ENABLED	false	Enable trusted-server token minting and browser token rendering.
RENDER_TOKEN_SECRET	unset	Dedicated hexadecimal sealing secret, at least 32 characters; required when tokens are enabled.
RENDER_TOKEN_SECRET_PREVIOUS	unset	Previous sealing secret accepted during rotation; remove after outstanding token expiry.
RENDER_TOKEN_MAX_TTL_MINUTES	60	Maximum token lifetime, 1-1440 minutes.
RENDER_TOKEN_MAX_RENDERS	100	Maximum render budget per token, 1-100000.

Keep sealing secrets on the server and separate from sandbox-token secrets. Origin-bound tokens require a matching Origin header on every authenticated request, including metadata. See embedding/token deployment constraints.

AI assistance and import settings

Variable	Default	Purpose
ANTHROPIC_TOOL_CHOICE	auto	Use automatic or forced tool selection for structured AI responses.
AI_EDITOR_ASSIST_ENABLED	true	Enable AI edit, expression and sample-data assistance when a provider key exists.
AI_TEMPLATE_IMPORT_ENABLED	true	Enable AI-assisted PDF/image import when a provider key exists.
AI_IMPORT_MAX_TOKENS	8192	Maximum provider output tokens for AI import.
IMPORT_ENABLED	true	Enable deterministic legacy-report JSON imports, independently of AI document imports.
IMPORT_MAX_SOURCE_BYTES	4194304	Complete legacy-import JSON body limit, 1024–4194304 bytes.
IMPORT_MAX_BYTES	10485760	Maximum document upload bytes.
IMPORT_MAX_PDF_PAGES	10	Maximum pages accepted for PDF import.

Shipped Prometheus alert rules and Grafana dashboards cover render admission, traffic, failures, licence state and scheduling. See Kubernetes observability configuration.

PDF optimisation and deployment-owned seals

Variable	Default	Purpose
PDF_OPTIMIZE_TIMEOUT_MS	30000	Maximum optional qpdf compaction duration.
PDF_IMAGE_CACHE_MB	128	Byte-bounded content-addressed resized-image cache; zero disables storage.
PDF_SIZE_BUDGET_MB	unset	Deployment warning budget, overridden by request/template sizeBudgetKb.
PDF_SIGNING_KEYS_JSON	unset	Signing registry JSON; supports PDF_SIGNING_KEYS_JSON_FILE for a mounted secret.
PDF_SIGNING_TSA_URL	unset	Operator-selected RFC 3161 authority, including private authorities.
PDF_SIGNING_TSA_AUTH_HEADER	unset	Authority Authorization header; supports PDF_SIGNING_TSA_AUTH_HEADER_FILE.
PDF_SIGNING_TIMEOUT_MS	15000	Authority timeout and initial KMS key verification timeout.
PDF_SIGNING_CONCURRENCY	4	Maximum concurrent signing operations in the API process.

Reuse PDF_QPDF_PATH for compaction and encryption. Signing and image resizing run in the API process; workers receive no private keys. Registry entries have id, kind, tenants, templates (wildcards * and ?), and allowUtility (default false). Multitenant deployments require explicit tenant bindings. For PEM, provide keyPath, certChainPemPath (leaf first) and optional passphraseEnv. For P12, provide path and optional passphraseEnv. Named passphrases support either that environment variable or its _FILE counterpart, never both. Mounted files must be readable by the service user. Modern AES/PBKDF2 PKCS#12 and legacy bundles are supported.

[{"id":"company-seal","kind":"pem","keyPath":"/run/secrets/seal-key.pem","certChainPemPath":"/run/secrets/seal-chain.pem","passphraseEnv":"SEAL_PASSPHRASE","tenants":["acme"],"templates":["invoice-*","statement-*"],"allowUtility":false}]

AWS KMS entries use kind: "aws-kms", keyId, region, certChainPemPath, and algorithm: rsa-pkcs1-sha256, rsa-pss-sha256 or ecdsa-sha256. The API uses its normal AWS credential chain and needs kms:Sign for that key. This path sends SHA-256 digests with MessageType: DIGEST; commerce’s separate signing path is unchanged. Boot verifies a random challenge against the leaf certificate, signing KeyUsage and validity. Invalid registry configuration fails startup with a redacted message.

PDF assembly variable	Default	Purpose
BOOK_MAX_PARTS	50	Maximum parts in a rendered book.
PDF_UTIL_MAX_OPERATIONS	20	Maximum pipeline operations.
PDF_UTIL_MAX_RENDER_SOURCES	5	Maximum template-rendered pipeline sources.
PDF_UTIL_MAX_SPLIT_PARTS	500	Maximum split artifacts.

Invoice XML validation (Wave 67)

Variable	Default	Purpose
EINVOICE_VALIDATOR_URL	unset	Operator-run KoSIT daemon HTTP(S) URL. Caller requests cannot select this endpoint. No credentials in the URL; use a private service/network if required.
EINVOICE_VALIDATOR_TIMEOUT_MS	15000	Per-request validation timeout, 100–300000 ms. Redirects are rejected; XML reports are capped at 5 MiB.

Local generation and PDF attachment handling require no external service. validate: true fails with 503 when the daemon is absent/unreachable and 422 when KoSIT returns 406. Only the first twenty rule messages are returned. /capabilities.eInvoice.validator reports configuration presence, not daemon health. A minimal daemon command is java -jar validator-1.6.3.jar -s kosit/scenarios.xml -r kosit -D -G -H 0.0.0.0 -P 8080; mount the pinned XRechnung configuration with its resources. Bind it to the application’s private network. Our independent gate pins KoSIT 1.6.3 with XRechnung 3.0.2 configuration 2026-08-31, Mustang 2.26.0 and veraPDF 1.30.3. Download URLs and SHA-256 checks are in .github/workflows/einvoice-conformance.yml.

Invoice attachments are buffered in API memory before PDF dispatch. Existing render body, PDF-util input/total byte limits and tenant render admission still apply. Keep production input schemas bounded. Schedules use saved template sources. Signing remains the final operation, after invoice XML is embedded.

Invoice XML supplied through xmlPath or inline base64 is trusted caller content; the engine does not reconcile it with visual PDF totals. Generic attachment content is distributed without antivirus scanning; the filename extension denylist is the only executable-content guard.

SQL sources and input conversion

Variable	Default	Purpose
SQL_DATA_SOURCES_ENABLED	false	Register PostgreSQL, MySQL and SQL Server template sources.
SQL_DATA_SOURCE_MAX_ROWS	100000	Server ceiling on each query’s row count.
SQL_DATA_SOURCE_TIMEOUT_MS	30000	Query deadline, including connection lookup and acquisition (maximum 60000).
SQL_DATA_SOURCE_MAX_POOLS	50	LRU pool ceiling; each pool has at most five sessions and drains on shutdown.
SQL_DATA_SOURCE_ALLOWED_HOSTS	empty	Comma-separated exact hostnames and CIDRs; required with multi-tenant mode. Checked addresses are pinned while TLS verifies the original hostname.
DATA_INPUT_MAX_BYTES	5242880	Decoded CSV/XML/XLSX byte cap. XLSX inflated ZIP bytes are additionally capped at ten times this limit.
DATA_INPUT_MAX_ROWS	100000	Conversion row ceiling.
DATA_INPUT_MAX_CELLS	2000000	Conversion cell ceiling, including tabular headers.

SQL passwords require the same secret encryption as HTTP connections. Create dedicated read-only database users: any editor can read anything granted to a connection’s user. Storage database host/port/database combinations, including resolved aliases, are refused. ssl.mode=verify-full validates the certificate and hostname; require encrypts without validating a self-signed certificate. PostgreSQL and MySQL run read-only transactions; every driver rolls back, streams bounded rows, and cancels on timeout or row overflow. SQL Server read-only protection relies on least-privilege credentials and mandatory rollback. SQL drivers are optional, lazy-loaded dependencies bundled in the container.

Template promotion governance

Variable	Default	Meaning
PROMOTION_POLICY_JSON	{}	Object keyed by protected label. Policies accept approvals (1..20), approverRole:"admin", forbidSelfApproval, requireTestRun, and allowBreakGlass.
LABEL_SCHEDULE_POLL_SECONDS	15	Promotion applier interval, 1..3600 seconds, in file, PostgreSQL and SQL Server modes.

Example: PROMOTION_POLICY_JSON={"prod":{"approvals":1,"approverRole":"admin","forbidSelfApproval":true,"requireTestRun":false,"allowBreakGlass":false}}. Unprotected labels retain direct promotion. Enable named-user identity for approval; API keys/shared-key identities cannot approve. The labels:approve action belongs to the built-in admin role and can be granted by custom roles. Capabilities advertise protected labels and whether named-user approval is available.

Apply the PostgreSQL or SQL Server migrations before deploying; existing pointers seed label history at their update time. History and comments cascade with permanent template deletion. History is not subject to audit purge. Multiple API instances use transactional claims and five-minute leases. File mode additionally requires filesystem support for atomic writes and exclusive directory creation; its journal recovers incomplete governance writes on the next access. Do not manually edit sidecars while the service is running. Monitor pulp_engine_label_promotions_total{outcome} and failed history entries. Scheduled events preserve their expected pointer; reconcile a failed event and schedule a new one.

Delivery connections and e-signatures

Variable	Default	Purpose
DELIVERY_PRIVATE_HOST_ALLOWLIST	empty	Comma-separated exact SFTP/S3 host names or IPs allowed to reach private addresses. Operator configuration only; DNS is resolved once and pinned.
DELIVERY_AZURE_ENDPOINT_SUFFIX_ALLOWLIST	empty	Production Azure custom-endpoint host suffixes, using exact hosts or *.example.com. Production endpoints require HTTPS; development allows Azurite.
ESIGN_ENABLED	false	Enables PDF signature-envelope routes, callbacks and scheduled e-sign delivery. Requires postgres/sqlserver storage, configured PULP_ENCRYPTION_KEY (or KMS), and PUBLIC_API_URL.
PUBLIC_API_URL	unset	Externally reachable API origin/base path for provider callbacks. Production requires HTTPS. Does not replace PUBLIC_BASE_URL document links.
ESIGN_POLL_INTERVAL_SECONDS	0	Optional bounded provider-status reconciliation interval; 0 disables polling. Connect callbacks remain the primary signal.

Self-service portal

Variable	Default	Purpose
PORTAL_ENABLED	false	Enable the named-user report portal and run-only viewer permissions.
PORTAL_MAX_SUBSCRIPTIONS_PER_USER	10	Concurrently enforced cap on personal portal schedules.
OIDC_VIEWER_GROUPS	empty	Comma-separated OIDC groups mapped to viewer before the editor wildcard.

Apply portal metadata, user-email and schedule-source migrations before enabling. Configure the database archive and blob storage for runs, and SMTP plus the schedule engine for subscriptions. See portal guide.

SaaS catalogue and tenant policy

Setting	Default	Purpose
TEMPLATE_CATALOGUE_TENANT_ID	unset	Enables a shared published catalogue in this tenant. Requires multi-tenant PostgreSQL or SQL Server. Cannot be default, a reserved slug or the sandbox tenant.
TEMPLATE_CATALOGUE_LABEL	prod	Release pointer visible to customer tenants; catalogue HEAD stays private.
TEMPLATE_CATALOGUE_FALLBACK	false	Inherit missing tenant template keys during rendering and composition. Explicit catalogue browsing/forking works without inheritance.
TENANT_SELF_SERVICE_RETENTION_MIN_DAYS	1	Minimum positive retention a tenant administrator may set.
TENANT_SELF_SERVICE_RETENTION_MAX_DAYS	3650	Maximum self-service retention; must be at least the minimum.
ASSET_BUCKETS_JSON	unset	Operator map of bucket aliases to {bucket,region,endpoint?,pathStyle?}. At most 32 locations; unknown fields and inline credentials are rejected. Requires multi-tenant private S3 assets. Uses global S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY; tenant settings contain aliases only.

Run both database migration sets before starting the new API. Catalogue identity is fixed while template forks or shared-asset consumers depend on it. Do not store customer data in the catalogue: its published releases and assets are shared with all authenticated tenants. Author with an ordinary credential bound to the catalogue tenant, then promote the release label.

Tenant settings are JSON Merge Patch documents with optimistic revisions. Features can narrow global flags. Monthly quotas share the existing quota engine and retain operator/credential ceilings; a soft tenant limit cannot bypass a hard operator limit. For accurate admission across replicas, configure Redis. Memory counters are process-local and recover from retained successful usage; retained usage and Redis durability determine restart recovery, so these counters are operational limits rather than billing records. Production/test credentials and scheduled renders consume monthly quotas; previews and PDF transforms do not. Page limits stop subsequent admission after accounted output reaches the cap.

Settings use TENANT_STATUS_CACHE_TTL_MS; PATCH invalidates the serving process immediately, and other processes converge within that TTL. A render receipt retains its admitted quota policy for settlement/refunds. Retention overrides exclude the tenant from the global sweep before its own cutoff is applied. Operator 0 means indefinite retention. Artifact settings apply to newly archived documents; positive durations remain capped by the global document maximum and any existing per-tenant document maximum. Explicit requested TTLs may shorten retention; operator indefinite retention applies when no TTL is requested. Signed download-link lifetimes remain independently bounded.

Enterprise provisioning and tenant SSO

See SCIM provisioning and SAML and tenant IdPs. Apply the lifecycle, provider and credential migrations before enabling these features.

Variable	Default	Purpose
SCIM_ENABLED	false	Mount /scim/v2; requires Postgres/SQL Server and EDITOR_USERS_DB=true.
SCIM_DELETE_MODE	deactivate	deactivate preserves attribution and revokes sessions; delete removes the account.
SCIM_BEARER_TOKEN	unset	Single-tenant shortcut bound to default; at least 32 non-whitespace characters. Prefer tenant-scoped API-key management.
OIDC_CLAIM_SCIM_LINK	sub	Environment-provider claim matched to tenant-local SCIM externalId; use oid for Entra object IDs.
OIDC_LINK_BY_EMAIL	false	Explicitly allow unique tenant-local SCIM email linking. Stable external IDs are preferred.
OIDC_TENANT_GROUP_MAPPING_JSON	unset	JSON object mapping verified group names to tenant slugs. All referenced tenants must exist and be active at boot.
OIDC_REQUIRE_TENANT_GROUP	false	Reject environment OIDC login without a mapped tenant group. Multiple matching tenants always fail.
AUTH_PUBLIC_BASE_URL	unset	Canonical HTTPS API origin for tenant SSO; requires native user storage and shared OIDC_COOKIE_SECRET.
AUTH_STATE_STORE	auto	redis, memory, or auto (Redis when REDIS_URL is set). Use Redis across replicas; failures fail closed.
IDP_PRIVATE_ORIGIN_ALLOWLIST	unset	Operator-approved exact HTTPS origins for internal tenant IdPs. Does not disable TLS or DNS pinning.

OIDC email linking also requires the verified email_verified=true claim; SAML email values come from the signed assertion.

Governed AI routing and narratives (Wave 77)

AI stays disabled when no provider is configured. Existing ANTHROPIC_API_KEY deployments select anthropic and retain ANTHROPIC_MODEL as an alias. Set the following variables before starting the server; incomplete explicit provider configuration fails at boot. Cloud SDKs are optional dependencies imported only for the selected provider.

Variable	Default	Meaning
AI_PROVIDER	anthropic when ANTHROPIC_API_KEY exists, otherwise disabled	anthropic, bedrock, vertex, foundry, gateway
AI_MODEL	ANTHROPIC_MODEL or claude-opus-4-7	Logical default model
AI_ALLOWED_MODELS	AI_MODEL	Comma-separated permitted logical model IDs
AI_MODEL_MAP_JSON	{}	Logical IDs to deployment IDs/inference profiles; Bedrock otherwise adds anthropic.
AI_AUTO_TOOL_CHOICE_MODELS	empty	Additional model IDs requiring automatic tool choice
AI_BASE_URL	provider endpoint / legacy ANTHROPIC_BASE_URL	Explicit endpoint; HTTPS required in production; paths are preserved
AI_EXTRA_HEADERS_JSON	{} / legacy ANTHROPIC_CUSTOM_HEADERS	At most 50 string headers; values never enter application logs
AI_HTTPS_PROXY	unset	HTTP or HTTPS CONNECT proxy URL, without URL credentials
AI_CA_CERT_FILE	unset	PEM trust bundle for custom server/proxy CAs; certificate validation remains enabled
AI_BEDROCK_REGION	required for Bedrock	AWS region; SDK uses the AWS default credential chain
AI_VERTEX_PROJECT_ID	required for Vertex	Google Cloud project
AI_VERTEX_REGION	required for Vertex	Explicit region; SDK uses Application Default Credentials/workload identity
AI_FOUNDRY_RESOURCE	required for Foundry	Resource name, without the .services.ai.azure.com suffix
AI_FOUNDRY_API_KEY	required for Foundry	Foundry credential; supports _FILE
AI_GATEWAY_TOKEN	ANTHROPIC_API_KEY fallback	Gateway bearer token; supports _FILE; bearer mode suppresses direct API-key auth
AI_INFERENCE_GEO	unset	us or global; sent only to direct Anthropic, never cloud partners or gateways
AI_MONTHLY_TOKEN_BUDGET	unlimited	Nonnegative input + output + cache-write + cache-read tokens per UTC month
AI_PROMPT_RETENTION	metadata	none also removes failure raw-output snippets; full adds redacted textual prompts to audit; provider logs are disabled
AI_PROMPT_REDACT_PATTERNS_JSON	[]	Up to 20 {pattern,flags?,replacement} rules; patterns limited to 200 characters; worker execution is time and memory bounded
AI_PRICING_JSON	{}	Logical model prices {inputPerMTok,outputPerMTok,cacheReadPerMTok,cacheWritePerMTok}; no built-in prices
AI_NARRATIVE_ENABLED	false	Enable grounded render-time summaries and author preview
AI_NARRATIVE_MODEL	AI_MODEL	Must be in AI_ALLOWED_MODELS when enabled
AI_NARRATIVE_MAX_PER_RENDER	10	Root narrative limit, at most 10
AI_NARRATIVE_MAX_CONCURRENCY	4	Per-process narrative concurrency, at most 4
AI_NARRATIVE_TIMEOUT_MS	15000	Per-call deadline, at most 15 seconds; entire narrative phase is bounded to 30 seconds

Explicit AI_MODEL, AI_BASE_URL and AI_EXTRA_HEADERS_JSON take precedence over legacy aliases; setting headers to {} explicitly clears legacy headers. AI_EXTRA_HEADERS_JSON_FILE is supported. Proxy credentials in URLs are rejected. Gateway deployments may use either a bearer token or an Anthropic API key. The factory disables SDK internal retries, so repair attempts, usage rows and request budgets share the same visible call boundary.

A super-admin can narrow tenant ai.enabled, ai.features (an array of generate, rewrite, narrative), ai.allowedModels, ai.monthlyTokenBudget, ai.budgetMode and ai.promptRetention through the existing revision-checked tenant settings API. Tenant administrators cannot change these operator fields. Features/models intersect global policy; token ceilings use the smallest configured limit and prompt retention cannot become less private. Editing/expression helpers consume the rewrite policy; import/sample generation consume generate policy. Binary import attachments are not transformed by text regex rules.

Native Postgres and SQL Server persist each provider attempt in ai_usage; file storage has process-local counters and warns when a budget is configured. Keep usage history covering the UTC month when enforcing durable monthly budgets. Counters refresh from storage every 30 seconds and retain known local increments. RATE_LIMIT_STORE=redis shares increments across pods. Calls already in flight are not reserved and can overshoot a budget. Hard exhaustion returns 429 and Retry-After to the next UTC month; soft exhaustion allows calls with X-PulpEngine-AI-Budget-Remaining. Usage retention follows RENDER_USAGE_RETENTION_DAYS and tenant usageDays; expired narrative cache entries are also swept daily.

Cloud smoke tests require operator accounts and model access. Before deployment, follow the provider checklist in the Wave 77 runbook. The region shown in capabilities is the requested API region. Inference-profile routing can use other regions: inspect the selected profile and provider logs rather than treating this field as a residency guarantee. AWS routing documentation.

Queue and hot-folder intake

Intake runs inside the API and uses the shared render-job pipeline, including template hooks, finishing, the batch render lane, quota accounting and mode: "ingest" usage rows. It supports PDF, DOCX, HTML, CSV, XLSX and PPTX (subject to the global and tenant PPTX gates). Built-in sources are sqs, folder and s3-prefix. AMQP, Service Bus and Kafka require a plugin adapter; first-party implementations are follow-ups.

Variable	Default	Purpose
INGEST_ENABLED	false	Master switch; disabled startup imports no adapter or intake bootstrap.
INGEST_SOURCES_JSON	unset	JSON array of at most 100 source definitions, bounded to 2 MiB.
INGEST_SOURCES_JSON_FILE	unset	Explicit file alternative; mutually exclusive with the inline array.
INGEST_MAX_IN_FLIGHT	2	Global intake cap; choose a value below available batch renderer capacity.
INGEST_INLINE_MAX_BYTES	131072	Maximum document bytes for a base64 inline reply; maximum setting 524288.
INGEST_MAX_MESSAGE_BYTES	1048576	Envelope and referenced JSON byte limit, at most 1 MiB.
INGEST_PAUSE_HIGH_WATERMARK	0.8	Stop pulling when active plus queued rendering reaches this fraction of admission capacity.
INGEST_RESUME_LOW_WATERMARK	0.5	Resume below this fraction; must be lower than the high watermark.
INGEST_SHUTDOWN_GRACE_MS	30000	Stop pulling and drain callbacks before aborting and releasing remaining claims.
INGEST_LEASE_MS	120000	Renewable idempotency and message visibility lease; 10000–43200000 ms.

Enable IDEMPOTENCY_ENABLED=true with IDEMPOTENCY_STORE=database on PostgreSQL or SQL Server, or IDEMPOTENCY_STORE=redis with REDIS_URL. Process-local memory is rejected for enabled intake. File storage with Redis has durable intake receipts and filesystem/S3 artifacts, but its render-job history remains process-local. Native database modes retain job history. Completed receipts use IDEMPOTENCY_TTL_HOURS (24 hours by default); retain output manifests and artifacts for at least your producer retry horizon. Redelivery after receipt expiry can run the renderer again, while an existing artifact retains its first bytes and version.

Each source has id, type, mandatory tenantId, optional enabled (true), maxInFlight (1), allowedTemplates, defaults: {format,label}, output, reply and a connection object. A source ID and its tenant binding must remain stable for retries. Source definitions are operator configuration; messages cannot set a tenant or output bucket. allowedTemplates and current tenant feature/delivery policies are checked again before replay. Archived or unavailable tenants pause their sources. Use a distinct source ID when changing tenant ownership.

[
  {
    "id": "finance",
    "type": "sqs",
    "tenantId": "acme",
    "maxInFlight": 1,
    "allowedTemplates": ["invoice"],
    "output": {"type": "s3", "bucket": "invoice-output", "prefix": "finance/"},
    "reply": {"type": "sqs", "queueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/finance-replies"},
    "connection": {
      "region": "us-east-1",
      "queueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/finance-input",
      "bucket": "invoice-input",
      "prefix": "incoming/",
      "dataPrefix": "data/",
      "visibilityTimeoutSeconds": 120,
      "waitTimeSeconds": 20
    }
  }
]

Use the AWS default credential chain, including IAM roles. Alternatively reference environment names with accessKeyIdEnv, secretAccessKeyEnv and optional sessionTokenEnv; each named secret supports its _FILE alternative. Inline passwords, tokens and credentials are rejected with redacted configuration errors. Operator-owned endpoint is optional; s3Endpoint can point object storage at a separate compatible endpoint. forcePathStyle is useful for MinIO. These trusted operator endpoints may address private infrastructure; envelope URL references and reply webhooks always use the pinned-DNS SSRF guard. Plugin connection secrets must also use named *Env or *File references.

The versioned envelope is ingest-message.v1.json. Supply a stable idempotencyKey, template, format and either data or dataRef. A reference is JSON with type: "url", url and optional headers, or type: "s3" and key. S3 data stays in the source’s configured bucket and dataPrefix (the input prefix by default); reserved control paths and traversal keys are rejected. CSV/XLSX input conversion is separate from this JSON envelope. Use either a pinned version or a label; source defaults fill absent selectors.

An envelope can select output: {type:"reply-inline"} or {type:"s3",key:"{template}/{date}/{correlationId}.pdf"}. S3 output stays in the configured output bucket/prefix and any narrower tenant delivery policy. Key tokens are limited to {template}, {date} (UTC enqueue date) and {correlationId}; values are escaped as path segments, with stable event identity as the correlation fallback. The default suffix follows the selected format. A durable manifest pins the first resolved output key across retries and date changes. Conditional creation and ownership metadata preserve the first artifact and its version; another message cannot overwrite that key. S3 output refuses stores that fail the conditional-write probe. Keep output and input prefixes/buckets separate and preserve _pulp_ingest/ (or your configured markerPrefix).

Replies contain eventId, sourceId, broker messageId, optional correlationId, template/version/format, output metadata or base64, duration and a sanitized failure code. They never contain tenantId. Reply delivery is at least once; consumers must deduplicate the stable eventId. Output is persisted, then the completed receipt, then the reply, and acknowledgement comes last. Permanent failures publish a failed reply with deadLettered: true and cache it before dead-lettering; cached poison is never rendered or republished. A crash between publishing and caching a failed reply can duplicate that reply. Temporary transport/render saturation failures release the claim and delay redelivery; quota exhaustion backs off until reset. Reply delivery failure always leaves the original message retryable, including a rejected webhook.

A reply destination is {type:"sqs",queueUrl} or {type:"webhook",url,secretEnv}. Webhooks use the existing timestamped HMAC signature and job ID headers, with eventId as the job ID. Input/reply/DLQ queue URLs must differ. FIFO sends use stable message deduplication identities. SQS input requires a redrive policy or explicit connection.deadLetterQueueUrl; prefer a sufficiently high redrive count for temporary outages. A native SQS redrive policy counts all receives, including transient retries, so broker redrive may still move repeatedly failing transient work to the DLQ. The adapter never deliberately marks transient failures as poison.

SQS also accepts aws:s3 ObjectCreated:* records from its configured input bucket/prefix. The object must contain an envelope; versioned notifications fetch the specified version. Every child record must settle before the parent receipt is deleted. Control markers and tagged engine output are ignored. Configure S3 event filtering for the input prefix. Amazon SQS currently allows 1 MiB bodies, increased in August 2025; inline base64, JSON framing and metadata all consume that budget. The adapter checks the final serialized reply size. Visibility is extended while processing but cannot exceed 12 hours from receive; duplicate broker delivery remains possible. See AWS payload limits and AWS visibility guidance.

For IAM, scope resources to the configured queues and prefixes. This example shows the operations; replace account, region, bucket and prefix placeholders:

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect":"Allow","Action":["sqs:ReceiveMessage","sqs:DeleteMessage","sqs:ChangeMessageVisibility","sqs:GetQueueAttributes"],"Resource":"arn:aws:sqs:REGION:ACCOUNT:finance-input"},
    {"Effect":"Allow","Action":"sqs:SendMessage","Resource":["arn:aws:sqs:REGION:ACCOUNT:finance-replies","arn:aws:sqs:REGION:ACCOUNT:finance-dlq"]},
    {"Effect":"Allow","Action":"s3:GetObject","Resource":"arn:aws:s3:::invoice-input/*"},
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::invoice-output/finance/*","arn:aws:s3:::invoice-output/_pulp_ingest/*"]},
    {"Effect":"Allow","Action":"s3:DeleteObject","Resource":"arn:aws:s3:::invoice-output/_pulp_ingest/probes/*"}
  ]
}

Versioned S3 notifications additionally require s3:GetObjectVersion on the input prefix. Bucket polling needs s3:ListBucket restricted by s3:prefix, plus Get/Put on its control-marker prefix and Delete on probe objects. Configure the input queue’s resource policy to allow S3 notifications only from the intended bucket and account. For native redrive, configure the DLQ redrive-allow policy; SendMessage on a DLQ is only needed for the explicit adapter destination. KMS-encrypted queues/buckets also need the applicable key permissions.

A folder connection uses absolute root, mandatory stable unique per replica instanceId, optional settleMs (2000) and pollIntervalMs (1000). Do not run two live replicas with the same instance ID. Producers write a temporary file and atomically rename it to inbox/name.json; close it before exposing it. The root must belong to one logical source/tenant and be writable only by trusted producers/operators. Symlinks are rejected. Layout:

root/
  inbox/                 # producers drop completed .json envelopes
  processing/instance/   # atomic input claims and durable metadata
  outbox/                # immutable document and eventId.reply.json
  processed/             # acknowledged inputs
  failed/                # poison inputs plus .error.json sidecars
  .pulp-ingest/           # source binding, artifact archive and manifests

Files younger than settleMs, non-JSON files and *.tmp/*.partial files are skipped. Atomic rename prevents competing replicas claiming the same input. Startup recovers only this instance’s abandoned claims, preserving message identity; graceful stop returns unfinished claims to inbox with metadata. Output uses hard links to atomically expose complete artifacts and preserve the first bytes, so the root must support same-volume atomic rename and hard links. Startup probes hard-link support. Test these semantics on the exact SMB/NFS server and mount options; polling alone does not establish them. Keep the private archive/manifests when cleaning public outbox files. Application-owned sidecars, archives, inbox/processed/failed data and S3 markers have no automatic retention purge in v1; remove them together only after the retry horizon has elapsed. Local outbox is the default reply/output destination; explicit reply or S3 output configuration overrides it.

An s3-prefix connection uses bucket, input prefix, pollIntervalMs (5000), leaseMs (120000), optional markerPrefix (_pulp_ingest/) and singleReplica (false). It polls .json inputs, keeps the original objects, and stores completed/failed receipt markers. Replacing an object with a new ETag creates a new input identity. Startup verifies both If-None-Match creation and If-Match renewal. If these are not honoured, it requires a Redis idempotency backend as a shared fencing fallback, or an explicit operator promise of singleReplica:true. The latter is safe only with exactly one active poller for that logical source. S3 output still requires conditional writes even when polling uses fallback leases. The engine heartbeats at half the smaller of the engine and bucket leases.

Administrative endpoints are GET /admin/ingest/sources (paginated), POST /admin/ingest/sources/:id/pause and /resume (204). They require admin scope and resolve the request’s tenant; super-admins supply the usual tenant header. Unknown or foreign sources return not_found. Responses contain only public status/counters; pause state is local to the API replica and resets on restart. For a lasting pause on every replica set enabled:false in the shared deployment config. Metrics are pulp_engine_ingest_messages_total{source,outcome}, pulp_engine_ingest_in_flight{source}, pulp_engine_ingest_source_paused{source}, pulp_engine_ingest_duration_seconds and pulp_engine_ingest_lag_seconds. Audit events use ingest_message and actor ingest:<sourceId>; no envelope bodies or secret values are logged. Lag is measured from broker enqueue/object modification time, and counters count deliveries, including replay/retry.

Plugin adapters register registerIngestSource(type,factory). The factory receives a frozen tenant-bound config, logger and named-secret reader after stores/jobs exist, and returns start, pause, resume, stop controls. Built-in names are reserved. Stop must return promptly after cancelling pulls; the engine drains render callbacks. Adapter callback promises and ack/nack/lease operations must settle, and adapters must honour pause and per-source concurrency. Plugin reply destinations use the same configured SQS/webhook publishers. The mock plugin context records source factories for plugin contract tests.

Optional Office converter

Set OFFICE_TEMPLATES_ENABLED=true and preserve TEMPLATE_SOURCE_DIR (default ./template-sources) or configure a private TEMPLATE_SOURCE_BUCKET. Compose mounts a separate private source volume. Enable PDF conversion with OFFICE_PDF_CONVERTER_URL=http://gotenberg:3000 and run docker compose –profile office up -d. The pinned Gotenberg 8.37.0 LibreOffice service has no published port and uses an internal Docker network without egress. Download-from is disabled and outbound LibreOffice URL access is denied. Filled packages have external relationships removed before conversion. OFFICE_PDF_TIMEOUT_MS defaults to 60000. Keep an equivalent no-egress network policy outside Compose. Include private source storage in backups. Database collection uses an extra pool connection while coordinating tenant source mutations; reserve pool capacity for metadata transactions. See Office templates.

OFFICE_TEMPLATE_MAX_BYTES defaults to 20971520 bytes and bounds Office uploads and native filling. Configure it consistently across API replicas.