Pulp Engine — Data Residency & GDPR Deployment Guidance

This document is for operators deploying Pulp Engine into environments with data-residency or GDPR obligations. It inventories the personal data Pulp Engine stores, explains how residency is achieved (single-region deployment), and gives a playbook for DSAR / erasure / portability requests.

Pulp Engine is operator-managed: the operator controls the infrastructure, the database, and the asset store. Pulp Engine itself introduces no third-party data flows unless AI template generation is enabled (§ 6).

1. Personal data inventory

All personal data stored by Pulp Engine lives in the operator’s Postgres + object store. There is no PulpEngine-operated cloud component.

Data	Storage	Retention control
Actor identity — named editor users (`id`, optional `displayName`, optional `email`)	`EDITOR_USERS_JSON` env var, plus Postgres `EditorUser` row on first OIDC login. See `apps/api/src/config.ts`.	Operator-managed: remove from env + delete row.
OIDC subject claim (`sub`) and `lastOidcLogin` timestamp	Postgres `EditorUser.oidcSub` + `EditorUser.lastOidcLogin`.	Operator-managed: deleted when the user row is deleted.
Editor session tokens (HMAC-signed, stateless)	Not stored. 5-part token `iat.expiry.tenantId.actor.sig` is fully self-contained and verified cryptographically. See `apps/api/src/lib/editor-token.ts`.	TTL via `EDITOR_TOKEN_TTL_MINUTES` (default 480, range 5–1440). Revoke all tokens cluster-wide by setting `EDITOR_TOKEN_ISSUED_AFTER` to a future timestamp.
Audit events — `actor`, `resourceType`, `resourceId`, `timestamp`, `details`	Postgres `AuditEvent` table.	`AUDIT_RETENTION_DAYS` (default 90) + background scheduler. Ad-hoc purge via `DELETE /audit-events?before=<ISO>` or `DELETE /audit-events?actor=<id>`.
Render usage — `actor`, `tenantId`, `timestamp`, `template`, `renderMode`	Postgres `RenderUsage` table.	`RENDER_USAGE_RETENTION_DAYS` + background scheduler.
Template content	Postgres or filesystem (depending on `STORAGE_MODE`). Templates themselves are not personal data, but user-supplied `data` passed into `/render` may contain personal data. Synchronous render inputs are not persisted — they flow through the render pipeline and are discarded. Two exceptions are listed in the rows below: schedule data sources and async-batch results.	N/A (not stored) for synchronous renders; see the exception rows.
Rendered output — synchronous renders (`/render/pdf`, `/render/docx`, …)	Returned in the HTTP response; not persisted server-side. If your integration stores them afterwards, that is out of Pulp Engine’s scope.	Downstream system’s responsibility.
Rendered output — async batch results (`POST /render/batch/async*`)	Persisted on Postgres deployments: completed result envelopes — gzipped JSON containing the full base64 rendered documents — are written to the job-result blob store (`JOB_RESULT_BLOB_DIR` on disk, or the private `JOB_RESULT_BLOB_BUCKET` on S3) so completed jobs survive restarts and stay pollable. File/SQL Server modes hold results in memory only.	`WEBHOOK_JOB_RETENTION_SECONDS` (default 3600 = 1 h) bounds both the job rows and the blobs. As of v0.84.0 an orphan-reconciliation sweep (boot + hourly) also removes blobs stranded by a restart, so retention is enforced even across crashes.
Scheduled-render artifacts	Each schedule execution writes its rendered document to a node-local temp file (`{tmpdir}/pulp-engine-schedule-{executionId}.{ext}`) for delivery/retry.	Unlinked ~5 minutes after successful delivery; a startup sweep removes stale artifacts older than `SCHEDULE_STALL_TIMEOUT_MS`. Node-local and short-lived, but it is rendered output at rest — factor it into disk-encryption posture.
Schedule definitions — `staticData` render inputs and `deliveryTargets` (which can contain recipient email addresses for email targets)	Postgres/SQL Server `Schedule` rows, for the life of the schedule. Execution rows store only delivery outcomes (`targetType`, delivered, attempts, error) — no recipient addresses.	Operator-managed: edit or delete the schedule. Include schedule rows in DSAR/erasure sweeps when recipients are data subjects.

IP addresses

IP addresses are intentionally not logged in audit events. The AuditEvent schema (apps/api/src/prisma/schema.prisma) has no IP column. This is a deliberate data-minimization posture, not a gap. Operators who need IP logging for security or compliance reasons should capture it at the reverse-proxy or LB access-log layer — those logs are outside Pulp Engine’s data model and under the operator’s retention policy.

2. Data residency

Residency = storage location. Pulp Engine’s durable data lives in two places:

The operator’s Postgres instance (DATABASE_URL).
The operator’s object store (S3_BUCKET) or filesystem (ASSETS_DIR).

Both are operator-provisioned. A single-region deployment = single-region data residency. There is no automatic cross-region replication, no PulpEngine-hosted state, and no telemetry that exfiltrates data to third parties.

Multi-region deployments

Pulp Engine does not natively replicate across regions. For multi-region obligations, run separate deployments per region:

EU deployment          US deployment          APAC deployment
├─ EU Postgres         ├─ US Postgres         ├─ APAC Postgres
├─ EU S3 bucket        ├─ US S3 bucket        ├─ APAC S3 bucket
└─ EU API pods         └─ US API pods         └─ APAC API pods

Tenants are routed to a region at provisioning time. In multi-tenant mode (MULTI_TENANT_ENABLED=true), tenant-to-region mapping is operator-controlled via API_KEYS_JSON and the OIDC OIDC_DEFAULT_TENANT configuration — each region has its own tenant registry.

Cross-region replication (read replicas in another region, fail-over) is possible at the database layer but not a Pulp Engine feature; it is the operator’s responsibility to ensure any such replication respects the same residency obligations as the primary region.

The procedures below assume a named-user deployment (EDITOR_USERS_JSON). Shared API keys (API_KEY_ADMIN, API_KEY_EDITOR) have no per-person attribution and do not fall under individual DSAR scope.

3.1 Right of access (Article 15 — DSAR)

Identify all data attributable to the subject’s actor identifier.

# 1. Audit trail
curl -H "x-api-key: $API_KEY_ADMIN" \
  "https://api.example.com/audit-events?actor=$ACTOR&limit=1000"
# Paginate via offset until total is exhausted.

# 2. EditorUser row (requires direct DB access — no API)
psql "$DATABASE_URL" -c \
  "SELECT id, display_name, email, oidc_sub, last_oidc_login FROM editor_users WHERE id = '$ACTOR';"

# 3. Templates/versions authored by the subject (createdBy field)
psql "$DATABASE_URL" -c \
  "SELECT key, version, created_at, created_by FROM template_versions WHERE created_by = '$ACTOR';"

# 4. Assets uploaded by the subject (createdBy field)
psql "$DATABASE_URL" -c \
  "SELECT id, filename, created_at, created_by FROM assets WHERE created_by = '$ACTOR';"

Collate the output and deliver to the subject within your statutory deadline (30 days under GDPR).

3.2 Right to erasure (Article 17)

Revoke active sessions. Set EDITOR_TOKEN_ISSUED_AFTER=<now> on all API pods. This invalidates every existing editor token cluster-wide on the next request.
Remove the identity. Delete the subject from EDITOR_USERS_JSON and remove the Postgres row:
```
psql "$DATABASE_URL" -c "DELETE FROM editor_users WHERE id = '$ACTOR';"
```

Erase the audit trail for that actor.

curl -X DELETE -H "x-api-key: $API_KEY_ADMIN" \
  "https://api.example.com/audit-events?actor=$ACTOR"
# Returns: { "deleted": <count> }

(New in this release — see apps/api/src/routes/audit-events/audit-events.routes.ts.)

Preserve or anonymize template/asset createdBy — operator decision. GDPR allows retaining the createdBy attribution on business records where there is a legitimate interest in version provenance; if erasure is required, null out the column via a direct DB update:
```
psql "$DATABASE_URL" -c \
  "UPDATE template_versions SET created_by = NULL WHERE created_by = '$ACTOR';
   UPDATE assets            SET created_by = NULL WHERE created_by = '$ACTOR';"
```
IdP deprovisioning. If using OIDC, remove the subject from the IdP as well — Pulp Engine does not auto-deprovision when the IdP does; it only auto-provisions on login.
Rendered-output stores. If the subject’s data was rendered via async batch within the last WEBHOOK_JOB_RETENTION_SECONDS, result blobs containing those documents may still exist in the job-result blob store — they age out within that window (≤ 24 h by config cap; 1 h default). For immediate erasure, delete the matching {tenantId}/{jobId}.json.gz objects directly. Check schedule rows for the subject’s email in deliveryTargets and edit/delete those schedules.
Record the erasure. Your compliance process should document the request, the fields erased, and the date.

3.3 Right to portability (Article 20)

Use a super-admin to export the tenant’s logical data:

pulp-engine tenant export acme --out acme.zip --api-url https://api.example.com --api-key "$SUPER_ADMIN_KEY"

The ZIP contains selected templates, asset binaries, schedules, users without credentials, audit and usage. Secrets are redacted. Tenant export covers the whole tenant; minimize the archive before returning a subject-specific subset. Export audit separately by actor when needed.

3.4 Right to rectification (Article 16)

Update the EDITOR_USERS_JSON entry (displayName / email) and restart the API pods, or update the Postgres row directly. Historical audit events are not rewritten — they record what was true at the time of action; that is the intended behaviour of an audit trail.

3.5 Tenant offboarding

Export and verify the ZIP before archiving. Wait TENANT_PURGE_GRACE_HOURS (24 by default), inspect the purge preview, then run pulp-engine tenant purge acme --confirm acme --wait. Keep the completed operation’s per-table and binary report. Purges resume after interruption. A tombstone permanently reserves the slug; bound credentials return 410. Default and sandbox tenants are protected.

Audit is retained by default until AUDIT_RETENTION_DAYS expiry; usage is erased by default. Use --no-retain-audit for stricter erasure. Lifecycle audit events and operation records remain as evidence. Private operation reports retain double-hashed credential fingerprints to reject erased keys with 410, without retaining authentication secrets; fingerprints are never exported or exposed through operation responses. Assess retained evidence against your organization’s policy.

Backups, external read replicas and noncurrent S3 versions need separate expiry procedures. Record the retention window, configure noncurrent-version lifecycle rules, and replay completed erasures after a restore before serving traffic.

4. Retention controls summary

Control	Default	Range	Effect
`AUDIT_RETENTION_DAYS`	90	any positive integer	Background scheduler purges audit events older than this.
`AUDIT_PURGE_INTERVAL_HOURS`	24	1+	How often the scheduler runs.
`RENDER_USAGE_RETENTION_DAYS`	90	any positive integer	Background scheduler purges render-usage rows.
`EDITOR_TOKEN_TTL_MINUTES`	480	5–1440	How long a minted editor token is accepted.
`EDITOR_TOKEN_ISSUED_AFTER`	unset	ISO 8601	Hard revoke: all tokens issued before this timestamp are rejected.
`WEBHOOK_JOB_RETENTION_SECONDS`	3600	60–86400	Bounds async-batch job rows and their result blobs (full rendered documents) on Postgres deployments. The v0.84.0 orphan-reconciliation sweep enforces it across restarts.
`JOB_RESULT_BLOB_STORE` / `_DIR` / `_BUCKET`	filesystem	—	Location control (not retention): where async-batch result blobs live — keep it in-region alongside the database for residency.
Scheduled-artifact cleanup	built-in	—	Schedule execution artifacts in `tmpdir` are unlinked ~5 min after delivery + swept at startup (`SCHEDULE_STALL_TIMEOUT_MS`).
Object-store lifecycle rules	operator	any	Unreferenced asset binaries — configure bucket lifecycle if needed.

Set these consistent with your compliance posture. AUDIT_RETENTION_DAYS=0 disables the purge scheduler (not recommended).

5. Tenant isolation

When MULTI_TENANT_ENABLED=true, every durable row is stamped with tenantId and every read/write enforces it. See tenant-isolation-guarantees.md for the contract and current gaps.

6. Subprocessors

The only subprocessors are those the operator chooses:

The Postgres provider (self-hosted, AWS RDS, GCP Cloud SQL, Azure Database, etc.).
The object-store provider (self-hosted MinIO, AWS S3, R2, GCS, etc.).
The IdP (Okta, Auth0, Entra, Keycloak, etc.) if OIDC is enabled. | Provider | Selected processor / configuration | | — | — | | anthropic | Direct Anthropic API; optional AI_INFERENCE_GEO; review the operator’s Anthropic agreement. | | bedrock | AWS Bedrock, AI_BEDROCK_REGION; inference profiles may route across regions. AWS routing guidance. | | vertex | Google Cloud, explicit project and region, ADC/workload identity. Provider data-handling guidance. | | foundry | Microsoft Foundry, named resource and key. Provider setup. | | gateway | The operator’s gateway and its downstream processors; Pulp Engine cannot infer their residency from the URL. |

All AI remains opt-in. Generation/import/author-assistance prompts go to the selected provider. Render-time aiNarrative additionally sends its static instruction and explicit evaluated fact values/descriptions, never the whole render payload. Cloud API region labels do not by themselves certify processing residency or retention; verify the chosen model/profile and provider account policies.

Pulp Engine itself operates no hosted service and introduces no additional subprocessors.

7. Known caveats

Named-user mode is required for per-person GDPR workflows. If you use only shared API_KEY_* credentials, audit events have no actor attribution and individual DSAR / erasure cannot be scoped. Deploy with EDITOR_USERS_JSON (and OIDC, ideally) for compliance-grade attribution.
Backups contain personal data. Retention and residency obligations apply to backup artifacts as well. Store backups in the same region, apply the same retention policy, and erase backed-up copies when an erasure request is completed (or document a reasonable delay window, typically ≤ 30 days).
Audit-by-actor erasure extends to the store layer only. If you have WAL archives or DB snapshots, those still contain the erased rows until the WAL retention window expires.
SCIM deprovisioning. Native database deployments can receive automatic user deactivation through tenant-scoped SCIM. The default preserves attribution and revokes sessions; permanent deletion is optional. See SCIM provisioning.

Rendered documents and signed links

The optional archive stores complete rendered output, filenames, caller metadata, template/execution references, actor/request identifiers, hashes, and download counts. Rendered output and metadata can contain personal data. Both the metadata database and the private filesystem/S3 archive must remain in the operator’s selected region; include them in access control, encryption, backup, restore, and erasure procedures.

Logical expiry defaults to seven days and is capped by the global/tenant retention configuration. Scheduled output defaults to seven days. Expired output and exhausted download caps stop serving immediately. Periodic purge removes bytes first, then rows; failed deletion retries later, and orphans older than one hour are swept. Exhausted caps have a 60-second physical deletion grace for already issued S3 URLs. Backups and provider failures require separate retention handling.

For an erasure request, find the relevant tenant’s documents through template/date/metadata filters, delete each with DELETE /documents/{id}, and handle backups under the operator’s erasure policy. Actor erasure from audit logs does not erase document content. Revoking links stops access through outstanding API signed URLs but does not erase bytes; a previously issued direct S3 URL can remain usable for at most 60 seconds. Tenant purge includes document metadata and current private document blobs.

Signed URLs are bearer credentials. Keep them out of third-party analytics, referrers, and public logs. Built-in request logging redacts the sig query value; custom logger instances and infrastructure access logs must apply equivalent redaction. Document audit events retain operation, document/tenant identity, format, and size, without signed URLs or content. Configure audit retention independently.

Render diagnostics and event payloads

Database render logs now persist success/failure status, error codes/messages, resolved template version/label, request/correlation IDs and integrator metadata. They never store render input data. Metadata and expression messages can still contain personal information. Use opaque business identifiers, avoid secrets and direct PII, and treat metadata as visible to every tenant admin. Set RENDER_LOG_ERROR_MESSAGES=false to omit messages from new log rows and render.failed events. Existing records are not rewritten by that setting. Set RENDER_USAGE_RETENTION_DAYS to purge both success and failure records; unset means indefinite retention. Preview failures are excluded by default.

Event subscriptions store signing secrets (and a previous secret during rotation). Restrict database/backup access. Durable delivery rows store exact signed JSON including event metadata, actor and correlation ID. Terminal rows are purged after EVENT_WEBHOOK_DELIVERY_RETENTION_DAYS (default 30); queued work is retained. Deleting a subscription cascades its delivery history. Data sent to a receiver falls under that receiver’s own retention policy. Configure only authorised destinations and include them in your data-flow inventory.

Stored credential encryption and audit feeds

Opt-in AES-256-GCM protects persisted editor keys, schedule header/webhook/PDF secrets, batch callback secrets and event-webhook signing keys. It does not encrypt arbitrary document data, URL query/userinfo credentials or all database columns. Configure storage/backups separately for full encryption at rest. Keep encryption key backups separately, and decrypt/backfill successfully before downgrade. See deployment guide.

Audit export omits tenantId and is scoped by the caller’s effective tenant. Operator-configured SIEM forwarding includes tenant IDs across all partitions; apply equivalent access/retention controls at the collector and include the collector in your data residency assessment. API-key actors identify the credential rather than a person. User governance events contain role/change metadata without keys or key hints.

Enterprise identity adds tenant-owned SCIM groups/members and identity-provider configuration. Tenant purge removes those rows and provider secrets before user records. Portable tenant bundles exclude provider configuration, secrets and group membership; supported user lifecycle/profile fields remain in explicitly selected user exports. SSO assertions are processed in memory and are not written to audit logs. Lifecycle audit entries identify their source without storing assertion XML, credentials or full claims.

AI accounting and narrative retention (Wave 77)

Native ai_usage stores tenant-scoped call metadata, actor/request/template identifiers, model/provider and four token dimensions. It contains no prompts or fact values. /usage/ai.csv supports a tenant export for access requests. AI usage is operational billing history and is excluded from portable template/tenant bundles; use the usage export explicitly. Its purge follows render-usage retention, including tenant overrides. File-mode counters are process-local.

AI_PROMPT_RETENTION=none removes raw failure-output snippets. metadata retains existing bounded failure diagnostics without storing prompts. full adds redacted textual prompts to audit events; binary attachment bytes are not copied to audit. Tenant policy can only narrow the global choice. Regex redaction happens before egress; it does not change provider-side logging or retention settings.

Narrative cache holds generated text and model/usage metadata, which can contain personal data derived from facts. Native cache is tenant-scoped and expires by TTL; file cache is a non-durable 1000-entry LRU. Cache keys contain hashes, not plaintext facts. ai_narrative_generated audit events record hashes, node/template IDs, verification and status; fact values are included only for full retention and are redacted. Cache hits do not create generation audit events. Tenant purge includes both AI tables (respecting the explicit retain-usage policy for the ledger), while narrative cache is always removed.

← Back to docs index