Kubernetes deployment

The chart in deploy/helm/pulp-engine supports Kubernetes 1.27+ and Helm 3. It runs API pods with RENDER_MODE=child-process. container, socket, in-process and renderWorkers.enabled are rejected: Docker-based dispatchers need a Docker daemon and are not Kubernetes-native. A remote Kubernetes render pool remains a separate feature.

Evaluation install

Build an API image or use a published release tag, then install:

helm install pulp deploy/helm/pulp-engine --namespace pulp --create-namespace \
  -f deploy/helm/pulp-engine/ci/eval.yaml --wait --wait-for-jobs --timeout 10m
helm test pulp --namespace pulp
kubectl --namespace pulp port-forward service/pulp-pulp-engine 3000:3000

After the first chart publication, the source chart can be replaced with oci://ghcr.io/troycoderboy/charts/pulp-engine --version 0.87.0. Publication happens through release.yml; committing the chart does not publish it.

The optional bundled PostgreSQL 16 and Redis 7 StatefulSets are evaluation services, without production backup, failover or TLS management. PostgreSQL has a PVC; evaluation Redis deliberately has no persistence. Default file-mode installs are single-replica and use a ReadWriteOnce data PVC. That strategy recreates the API on upgrades and can interrupt service.

The evaluation values contain development credentials. Replace them for any shared environment. Chart-generated credentials are reused using Helm lookup; offline helm template cannot look up previous credentials and generates new ones. Prefer existingSecret for GitOps and production. Never commit real credentials into values files.

Production configuration and secrets

Start with ci/production.yaml. Provision the database, Redis, object-store bucket and Kubernetes Secrets before installation. The production example references pulp-engine-api, pulp-engine-db and pulp-engine-redis; edit these names and the S3 endpoint/bucket/CORS values for your environment. pulp-engine-db contains DATABASE_URL (Postgres) or SQL_SERVER_URL (SQL Server); pulp-engine-redis contains REDIS_URL. The shared API Secret contains API_KEY_ADMIN, API_KEY_EDITOR, API_KEY_RENDER, and the configured signing/encryption, S3 and licence credentials. Every API and scheduler pod receives the same Secrets through envFrom.secretRef.

config accepts non-secret environment variables. Credential-shaped keys are rejected there. secrets supplies the chart-managed evaluation Secret; existingSecret substitutes a pre-created shared Secret. With bundled services, the chart generates DATABASE_URL and REDIS_URL. If you combine bundled services with existingSecret, supply those URLs yourself to match the bundled service names/passwords. Secret file mounts remain an operator-managed option; this chart uses shared environment Secrets.

Use database storage, S3 assets/results/documents and Redis rate limiting for multiple API replicas. The production example uses private S3 buckets for result blobs (including reusable datasets) and stored documents. Filesystem blob storage needs the same shared ReadWriteMany volume as filesystem assets; ephemeral per-pod storage cannot serve another replica’s blobs. Shared filesystem assets require an enabled ReadWriteMany claim. File storage cannot scale. The explicit allowInMemoryRateLimit escape hatch permits independent per-pod limits; it does not make those limits globally consistent. Pin image.digest when reproducible rollbacks matter.

Migrations, upgrades and rollback

External databases use a pre-install,pre-upgrade migration Job against the existing database Secret, before API rollout. Postgres runs Prisma migrate deploy; SQL Server runs the packaged migration script. Back up the database first and review migration compatibility before upgrade.

A bundled PostgreSQL first install uses a normal migration Job: a pre-install hook would run before the chart creates PostgreSQL and deadlock. API init containers wait until every packaged migration is complete. Bundled database upgrades use a pre-upgrade hook because the database already exists. helm --wait --wait-for-jobs checks first-install completion. Disabling migration.enabled makes schema preparation your responsibility.

helm upgrade pulp deploy/helm/pulp-engine -n pulp -f production-values.yaml --wait --wait-for-jobs --timeout 10m
helm rollback pulp PREVIOUS_REVISION -n pulp --wait --timeout 10m

Helm rollback does not reverse database migrations. Restore a compatible backup when required. Rotating an externally managed Secret requires restarting the deployments; chart-managed secret/config changes have checksum rollout annotations.

Hardening and render isolation

API, scheduler and migration containers run as UID/GID 1001, with a read-only root filesystem, all capabilities dropped, no privilege escalation and no mounted service-account token. /tmp is writable with a 1Gi limit; HOME and Chromium XDG paths point there. /data is the configured PVC or an ephemeral volume. A RuntimeDefault seccomp profile is the default. Set runtimeClassName to an installed gVisor/Kata class for stronger pod isolation.

The default sets PULP_ENGINE_DISABLE_SANDBOX=true. Chromium therefore runs without its own sandbox inside the hardened pod. NetworkPolicy and application SSRF controls provide additional boundaries. render.chromiumSandbox.enabled=true opts into the shipped Localhost seccomp profile and Chromium’s user-namespace sandbox. Install files/chromium-seccomp.json on every eligible node at the configured localhost profile path, or use createProfile=true with Security Profiles Operator already installed. The operator mode uses operator/<namespace>/<fullname>-chromium.json, following SPO profile paths. Migration Jobs use RuntimeDefault because they do not launch Chromium and can run before the optional profile resource exists. The profile is derived from Moby’s Apache-2.0 default and adds clone/unshare/setns allowances. Node kernels must allow unprivileged user namespaces; AppArmor on Ubuntu 24.04 can block them. Validate this option on your own node image before enabling it. It is not part of the default smoke test.

Probes, termination and capacity

Startup probes use /health every 5 seconds for up to 180 seconds; browser warmup precedes listening. Readiness probes use /health/ready every 10 seconds with a 3-second timeout. Liveness uses /health without dependency checks.

On SIGTERM, readiness returns 503 with draining: true, while liveness remains 200. The app waits SHUTDOWN_DRAIN_DELAY_MS (chart default 10000; application default 0) before closing. A 10-second preStop delay precedes SIGTERM. The default 90-second termination grace exceeds preStop + drain + the 65-second render-worker timeout; shorter combinations are rejected. Requests already accepted may complete; clients should retry 503 responses appropriately.

The defaults request 1 CPU and 2Gi RAM and limit memory to 4Gi for four active pages plus a queue of sixteen. Chromium can consume approximately 150-300 MB per page; measure real templates and adjust page capacity and memory together. The new capacity gauge exposes active plus queued admission slots, not browser memory capacity.

Enable either autoscaling (CPU HPA at 70%) or keda (Prometheus in-flight work plus CPU), with the relevant cluster components installed. KEDA requires a reachable Prometheus endpoint and matching service/namespace scrape labels. Scale-down waits 300 seconds and removes at most one pod per 120 seconds. PodDisruptionBudget applies to voluntary disruptions. Scale-in and rollouts can still fail processing async batch jobs; startup reconciliation records these failures rather than silently replaying them. A dedicated scheduler enables schedule execution and retention jobs only on its deployment; multiple replicas still require shared database/Redis/assets.

Network policy

networkPolicy.enabled limits API ingress on 3000 to ingressFrom selectors. Add ingress-controller, Prometheus and any client namespaces. Egress permits DNS, bundled database/Redis, and configured database/Redis/object-store/extra CIDR and port lists. Optional public HTTPS excludes private, loopback, link-local and shared address ranges. Disable it for isolated installations and specify required destinations, including SMTP, webhooks, OIDC and AI providers.

NetworkPolicy needs a CNI that enforces it; kind’s default networking does not prove enforcement. Policies cannot match DNS hostnames. Cilium FQDN policies are a follow-up for changing provider IP ranges. The Chromium child shares the pod network, so keep BLOCK_REMOTE_RESOURCES and ALLOWED_REMOTE_ORIGINS configured as well.

Images and observability

Release builds use native amd64 and arm64 runners. amd64 retains Puppeteer’s Chrome for Testing; arm64 uses Debian Chromium. Both use /usr/local/bin/pulp-chromium. Prisma includes the arm64 OpenSSL 3 engine. Native smoke renders must have identical extracted text and page counts before the release publishes combined API/worker/controller indexes. Per-platform SBOMs and Trivy reports and recursive cosign signatures are release artifacts. Different browser versions can still affect layout beyond that smoke fixture; validate customer templates before switching architecture.

Import deploy/observability/grafana/*.json into Grafana 10+, selecting the Prometheus datasource, namespace and job. Optional dashboard ConfigMaps support Grafana sidecar discovery. deploy/observability/prometheus-rules.yaml works without Helm; ServiceMonitor/PrometheusRule options require Prometheus Operator CRDs. Set the metrics bearer-token Secret if metrics authentication is enabled. Chart thresholds override error ratio, render p95, saturation, licence days and event-loop lag. No tenant identifiers are metric labels. The scheduling dashboard links to authenticated diagnostics; failure counters are not a DLQ depth gauge.

Verification

scripts/k8s/verify-chart.sh
PULP_IMAGE=pulp-engine:local scripts/k8s/verify-chart.sh --kind

The first command requires Helm, kubeconform, Node 22+ and Docker (or PROMTOOL_BIN). CRD schemas are vendored from pinned upstream releases. The kind test additionally needs kind, kubectl, Bash, curl and jq. It creates an isolated cluster/kubeconfig, loads the local image, installs evaluation PostgreSQL, waits for migrations and readiness, runs helm test and the existing PDF smoke test, and checks a 30-second pod deletion loop. A stable nginx proxy maps unavailable single-replica upstreams to 503. The test permits those 503s but rejects every other failure and directly observes the old pod’s draining readiness. It checks availability handling, not uninterrupted single-replica service. Resources created by the test are cleaned up.

Tool paths can be set with HELM_BIN, KUBECONFORM_BIN, KIND_BIN, KUBECTL_BIN and BASH_BIN. To reuse a task-owned kind cluster, set KIND_REUSE=true, KIND_CLUSTER and an isolated KUBECONFIG; only the test namespace is deleted afterward. Run this smoke manually before tagging; schema and rule validation also run in the release job before Helm publication.