Check Sheet Migration

Create quality checks, and the computed tables or joins they depend on, from a single spreadsheet. The check sheet is a normalized tabular template (Excel or CSV, one row per check or computed container) designed to be filled from a client's own check catalog, so a migration that starts life in a tracking spreadsheet ends as checks in Qualytics without a hand-written script.

Commands

Workflow

graph LR
    S[check sheet] -->|migrate plan| P[Offline summary and issues]
    S -->|migrate validate| V[Names, references, and SQL resolved against the target]
    S -->|migrate apply| Q[(Qualytics datastore)]
    Q --> A[Run artifacts: run.log, results.csv, yaml/]

1. Fill the sheet from the source catalog.
2. migrate plan catches structural problems without touching anything.
3. migrate validate --datastore-id N proves every name, reference, and SQL query resolves on the target. Still creates nothing.
4. migrate apply --datastore-id N creates computed containers first, waits for each one's profile, then imports the checks.

Every row problem is reported with its row number and check_id; rows with errors are skipped, never half-applied.

The check sheet

One .xlsx or .csv file. Column headers are matched case-insensitively with spaces and punctuation normalized, so Check ID, check_id, and CHECK-ID are the same column. A workbook with several tabs is read from the first tab by default; --worksheet picks another by name or position. Unrecognized columns are ignored with a single warning naming them.

Identity columns

Check columns

Rule properties

Required columns are validated per rule before anything runs: freshness requires value; existsIn requires fields, ref_container, and ref_field; aggregationComparison requires expression, comparison, ref_container, and ref_expression.

Computed container columns

A check row that targets a computed container declared in the same sheet just names it in container; apply orders the phases. Row order never matters between checks and containers. It matters only among computed containers themselves: a computed_join may read a computed table from the same sheet only if that table's row comes first.

Custom metadata columns

A column headed metadata:<key> writes <key> into every produced check's additional_metadata, key taken verbatim, casing and punctuation preserved. An empty cell means the key does not apply to that row, so one sheet can carry metadata:X for some rows and metadata:Y for others:

check_id,rule_type,container,fields,metadata:Business Domain,metadata:SLA Tier
100,notNull,orders,order_id,Treasury,
200,unique,invoices,invoice_id,,gold

legacy_check_id is reserved (it is set from check_id); a metadata: column naming it is an error.

Plan

Validate and summarize the sheet offline. No authentication, no network.

qualytics migrate plan --sheet week1.xlsx
qualytics migrate plan --sheet week1.xlsx --worksheet "Week 1" --show-checks

Problems are graded: errors (a missing required column, a duplicate check_id, an unknown kind) stop those rows from ever applying, while warnings (an unrecognized column, date logic with no explicit time zone conversion) apply but deserve a look.

Validate

Fail early. Everything plan checks, plus read-only resolution against target datastores: containers exist (or the sheet creates them), field names match the synced schema, every ref_container, ref_datastore, and ref_field resolves (each printed with the container it resolved to), join sources exist, and each computed container's SQL is checked against the source without creating anything. Exits non-zero on any problem.

qualytics migrate validate --sheet week1.xlsx --datastore-id 42

Validating against datastore 42 (read-only)...
  row 6 (770): SQL for 'orders_recon' validated against the source
  row 4 (326): ref 'CUSTOMER' → container 169 in datastore 43
  row 5 (211): ref 'REGION' → container 176 in datastore 43
  datastore 42: all references resolve

Validation passed.

If the instance itself cannot be reached (an expired sign-in, a connectivity problem), validate reports that once as an instance problem rather than flagging every computed row, so a real SQL mistake is never confused with an outage.

Apply

Create everything on the target. Two phases, in order:

1. Computed containers. Every specification is validated first; a validation failure stops the phase before anything is created. Containers are then created in declaration order, and the CLI waits for each container's own profile operation to finish before moving on, so dependent joins and checks always see profiled fields.
2. Checks. Container and field names are case-corrected against the synced schema, cross-references resolve to the right datastore, and checks are created or updated in place. Failures are reported per row and never stop the run.

# Preview: no changes, dependent checks count as creates
qualytics migrate apply --sheet week1.xlsx --datastore-id 42 --dry-run

# Apply
qualytics migrate apply --sheet week1.xlsx --datastore-id 42

# Re-apply after sheet edits, keeping checks a reviewer already activated
qualytics migrate apply --sheet week1.xlsx --datastore-id 42 --preserve-status

Updating existing computed containers

--on-existing update compares the sheet against the live definition and only sends real changes: an identical definition is reported unchanged, a description or metadata edit takes a light path that never re-profiles, and a genuine SQL change triggers validation, the update, and a fresh profile. A change that would drop fields carrying quality checks is refused by the platform; add --force-drop-fields to proceed (the affected checks are preserved and reactivate if the fields reappear), and update the sheet's dependent check rows to the new field names.

Run artifacts

Every real run leaves an audit folder (dry runs write nothing):

migrate-runs/20260917-150505/
├── run.log        # the full terminal transcript, timestamped per line
├── results.csv    # one row per check: check_id, created/updated/failed,
│                  # container, rule, status, a link to the check, and the
│                  # failure reason for failed rows
├── failures.log   # failure detail
└── yaml/          # the definitions exactly as applied, per datastore

results.csv is the receipt that goes back to whoever owns the source catalog: their ID, what happened to it, and where it now lives.

Cross-datastore references

ref_container never resolves globally. With ref_datastore filled, the name is looked up only in that datastore; without it, only in the row's own target datastore. A name that does not resolve in the scoped datastore is an error, never a guess, and names whose casing is ambiguous on the target (say Orders and ORDERS both exist) are rejected with both variants named.

check_id,rule_type,container,fields,ref_datastore,ref_container,ref_field
900,existsIn,STG_POSITIONS,PORTFOLIO_ID,Analytics Warehouse,portfolio_status,PORTFOLIO_ID

This row's check lives on STG_POSITIONS in the apply target, and its reference resolves inside the Analytics Warehouse datastore. migrate validate prints the resolved container for every reference so a wrong scope is visible before anything is created.

Idempotency

Checks are matched by legacy_check_id, the sheet's own check_id. Re-running apply after sheet edits updates checks in place rather than duplicating them, and two same-shaped checks with different IDs stay separate. Because a re-apply sets status from the sheet by default, use --preserve-status so checks a reviewer activated in the platform stay active.