Template test suites

Run dry-runs, document content assertions and snapshots before promoting a template. The harness supports stored versions and local definitions. CI and the editor enforce their own gates; the server does not require a passing test run for label promotion.

pulp-engine test ./tests
pulp-engine test ./tests --local ./templates --datasets-dir ./templates/datasets
pulp-engine test ./tests --update-snapshots
pulp-engine test ./tests --reporter html --artifacts-dir ./pulp-engine-test-results
pulp-engine test ./tests --reporter junit --output test-results.xml

--local without a directory uses the configured templates directory. It maps each fixture key to <dir>/<key>.json, sends the definition to preview routes without pushing it, and reports the version as local. An inline-route 404 explains PREVIEW_ROUTES_ENABLED=false. Render credentials suffice for pinned fixtures; shared dataset reads require editor or admin credentials. Exit codes remain 0 for success, 1 for test/fixture failure, and 2 for CLI configuration errors.

Fixtures

template: invoice
version: "1.0.0"
now: "2026-01-01T00:00:00Z"
tests:
  - name: shared scenarios
    input: {dataset: '*'}
    expect:
      dryRun: strict
      htmlSnapshot: __snapshots__/{dataset}.html
      pdfSnapshot:
        dir: __snapshots__/pdf/{dataset}
        pages: all
        thresholdPercent: 0.5
        pixelThreshold: 0.1
      pageCount: {min: 1, max: 3}
      contains: [Invoice, 're:Total\s+[0-9]+']
      notContains: [undefined]
      noUnresolvedBindings: true
      maxBytes: 500000
      tableRows: [{tableId: items, min: 1, max: 500}]
      docxSnapshot: __snapshots__/{dataset}.docx.txt
      xlsxSnapshot: __snapshots__/{dataset}.xlsx.json

Use templateFile: ../templates/invoice.json instead of version for an explicit local definition. Exactly one is required. Template files and input files must contain JSON objects. Input can be inline or {file: ./data.json}. The file and dataset forms are special only when they are the object’s sole key, so business data with a file property remains valid inline input.

input: {dataset: name} resolves one shared sample. '*' expands non-negative datasets into <case> [<dataset>] cases. {dataset} is substituted in snapshot paths. --datasets-dir reads the {meta,data} files created by pull --datasets without dataset API calls. Fixture and case now values pin @render.now; the harness otherwise uses 2026-01-01 for repeatable tests.

Expect option	Behavior
dryRun	ok (default), strict (also requires every binding), fail (negative test), or skip
htmlSnapshot	Normalized HTML baseline file
pdfSnapshot	Legacy first-page PNG string or multi-page object {dir,pages,thresholdPercent?,tolerancePixels?,pixelThreshold?}
pageCount	Exact positive integer or {min?,max?} range
contains, notContains	Literal text or re:<pattern> regex, checked after text normalization
textSource	html (default) or optional pdf extraction
maxBytes	Maximum PDF byte length
noUnresolvedBindings	Fails on binding warnings or when an older server omits diagnostics
tableRows	RFC 4180 CSV record count, minus the header, for each selected table
docxSnapshot	Normalized DOCX outline baseline
xlsxSnapshot	Normalized worksheet/cell JSON baseline

Every needed format renders at most once per case, including when several assertions need its output. Every content assertion runs and reports its failure. A failed dry-run stops that case before expensive rendering. HTML text strips tags, scripts and styles, decodes entities and collapses whitespace.

Multi-page PDF baselines and artifacts

Install optional raster dependencies in the test project:

npm install --save-dev pdf-to-png-converter pixelmatch pngjs
# Only needed for textSource: pdf:
npm install --save-dev pdfjs-dist

pages: all compares every page. An explicit list such as [1,3] rasterizes only those pages. Baseline directories contain page-001.png, page-002.png, and a .pages.json manifest recording the full document page count. Page count changes fail with both counts. --update-snapshots writes the selected pages and removes stale page PNGs, including pages removed by pagination changes. The legacy string form retains first-page behavior and tolerancePixels.

For the object form, both thresholds must pass: differing pixels must be within tolerancePixels when set, and the percentage must be within thresholdPercent (default 0). pixelThreshold controls pixelmatch colour sensitivity (default 0.1), independently of the allowed percentage. A page-three-only change produces artifacts only for page three:

pulp-engine-test-results/<fixture>/<case>/page-003.expected.png
pulp-engine-test-results/<fixture>/<case>/page-003.actual.png
pulp-engine-test-results/<fixture>/<case>/page-003.diff.png

A missing baseline produces actual/diff artifacts where possible. Use the same OS/container image, fonts, Chromium and pdfjs versions when generating and checking baselines. Anti-aliasing can differ across environments; percentage tolerance absorbs small rendering noise. Local preview HTML can differ from production HTML (for example asset inlining); keep their baseline sets separate.

Structural snapshots and format coverage

DOCX snapshots read document, header and footer XML from the ZIP. They retain paragraph styles and text (P[Heading1] Invoice), table dimensions and rows, image dimensions and page breaks. They omit volatile relationship IDs. XLSX snapshots retain sheet names, frozen panes, cell coordinates, values, types, number formats and formulas. Both use readable line diffs.

Format	Checks
PDF	Multi-page pixels, page count, maximum bytes, optional text
HTML	Normalized snapshot and text
DOCX	Structural outline snapshot
XLSX	Structural cell-grid snapshot
CSV	Per-table data record counts
PPTX	No dedicated snapshot/assertion support in this wave

Reports and CI

--reporter html writes a standalone index.html under the artifacts directory. It opens offline and links expected, actual and diff images side by side. Console output names artifact paths and all failed checks. JUnit includes [[ATTACHMENT|path]] markers in system-out.

Upload the whole artifacts directory, preserving its relative paths, and the JUnit XML even when tests fail. .github/workflows/template-ci.sample.yml demonstrates actions/upload-artifact with if: always() and promotion after a successful test job. Never update baselines automatically as part of a regression check; review changes before committing them.

Saved dataset checks in the editor

Datasets may store the content subset as assertions: page count, literal HTML contains/notContains, maximum PDF bytes, missing bindings and table row counts. POST /templates/:key/tests/run runs at most the configured dataset cap with concurrency two, skips negative/schema-stale cases, and returns check results plus testRunId. It never returns rendered bytes, input data or tenant IDs. Every run is audited. Its PDF helper shares tenant asset, resource trust and dispatch policy with the PDF handlers.

The editor’s Dataset details exposes these checks. When /capabilities.templateTests is true, the publish gate runs them against the unsaved template and blocks failed cases. Otherwise it uses the dataset validation preflight. The editor rechecks dataset revisions before publication. Server promotion does not enforce this verdict; a test-run ID is a trace breadcrumb, not authorization to promote.

PDF text and raster checks use compatible PDF.js 6 and pdf-to-png-converter 4 versions. These optional tools require Node 22.13 or later. Windows reserved dataset names such as con use __con.json on disk; the metadata and API name remain con.