Skip to content

Export and import full configuration

Goal

Treat the entire data quality configuration of a Qualytics datastore as code: connections, datastore settings, computed containers, computed fields, and quality checks. Export the lot to a Git-friendly folder, version it, then re-import to another instance (Dev, Staging, Prod) or back to the same instance after edits.

Permissions

Step Endpoint Role Team permission
Read connections GET /api/connections Member N/A
Read datastore GET /api/datastores/{id} Member Reporter
Read containers / fields / checks GET /api/containers, /api/quality-checks, etc. Member Reporter
Create / update connections POST / PUT /api/connections Manager N/A
Create / update datastores POST / PUT /api/datastores Manager (create), Member + Editor (update) Editor for updates
Create / update computed containers POST / PUT /api/containers Member Editor
Create / update quality checks POST / PUT /api/quality-checks Member Author

Mixed permissions

A config import that creates connections requires a Manager token; one that only updates checks doesn't. The simplest pattern is to use a Manager service account for full imports.

Prerequisites

  • The CLI is installed and authenticated.
  • For imports with secrets in connection files: every ${ENV_VAR} placeholder must be set in the environment or in a local .env file.

CLI workflow

graph LR
    Source[(Source datastore)] --> Export[config export]
    Export --> FS[Folder structure]
    FS --> Git[Git repo]
    Git --> Pull[git pull on target]
    Pull --> Dry[config import --dry-run]
    Dry --> Apply[config import]
    Apply --> Target[(Target datastore)]

1. Export

qualytics config export --datastore-id 1 --output ./qualytics-config

Resulting layout:

qualytics-config/
├── connections/
│   └── warehouse-prod-db.yaml
└── datastores/
    └── warehouse-prod/
        ├── _datastore.yaml
        ├── containers/
        │   └── high_value_orders/
        │       ├── _container.yaml
        │       └── computed_fields/
        │           └── full_name.yaml
        └── checks/
            └── orders/
                ├── isNotNull_customer_id.yaml
                └── satisfiesExpression_total_positive.yaml

Multiple datastores in one export:

qualytics config export --datastore-id 1 --datastore-id 2 --datastore-id 3

Selective resource types:

qualytics config export --datastore-id 1 --include connections,datastores,checks

2. Commit and review

git add qualytics-config/
git diff --stat
git commit -m "Snapshot of warehouse-prod"
git push

3. Preview the import to the target

export QUALYTICS_URL=https://prod.qualytics.io
export QUALYTICS_TOKEN=$PROD_TOKEN
export DB_HOST=prod-db.example.com
export DB_USER=qualytics_reader
export DB_PASSWORD=$PROD_DB_PASSWORD

qualytics config import --input ./qualytics-config --dry-run

4. Apply

qualytics config import --input ./qualytics-config

Behind the scenes

config import is dependency-ordered: connections → datastores → computed containers → computed fields → quality checks. Within each layer the CLI matches by name (or _qualytics_check_uid for checks) to decide create vs. update.

Phase Method Path Notes
Connections (create) POST /api/connections Per new connection.
Connections (update) PUT /api/connections/{id} Per changed connection.
Datastores (create) POST /api/datastores Connection ID resolved by name.
Datastores (update) PUT /api/datastores/{id}
Computed containers (create) POST /api/containers Followed by PUT for description/tags.
Computed fields (create) POST /api/computed-fields
Quality checks (create) POST /api/quality-checks Per check that doesn't exist on the target.
Quality checks (update) PUT /api/quality-checks/{id} Per check matched by _qualytics_check_uid.

Python equivalent

config export/import covers a lot of ground (folder walking, dependency ordering, secret resolution, idempotent matching). Re-implementing it in Python is rarely worthwhile. For automation that's already in Python, the cleanest pattern is to call the CLI as a subprocess:

import os
import subprocess

env = {**os.environ,
       "QUALYTICS_URL":   "https://prod.qualytics.io",
       "QUALYTICS_TOKEN": os.environ["PROD_TOKEN"],
       "DB_HOST":         os.environ["PROD_DB_HOST"],
       "DB_USER":         os.environ["PROD_DB_USER"],
       "DB_PASSWORD":     os.environ["PROD_DB_PASSWORD"]}

subprocess.run(["qualytics", "config", "import",
                "--input", "./qualytics-config", "--dry-run"],
               env=env, check=True)
subprocess.run(["qualytics", "config", "import",
                "--input", "./qualytics-config"],
               env=env, check=True)

For a hand-rolled subset (e.g., "I only want to push checks via Python"), see the Python example in Promote checks Dev to Prod.

Variations and advanced usage

Restrict by resource type on import

qualytics config import --input ./qualytics-config --include checks
qualytics config import --input ./qualytics-config --include connections,datastores

Per-environment overrides

The folder is environment-agnostic; the differences between Dev and Prod are entirely in environment variables:

# Dev
DB_HOST=dev-db.example.com qualytics config import --input ./qualytics-config

# Prod
DB_HOST=prod-db.example.com qualytics config import --input ./qualytics-config

If structural changes are needed per environment (e.g., extra checks only in Prod), consider keeping environment-specific YAML in qualytics-config/overrides/<env>/ and importing them separately on top.

Roll back

git revert the relevant commit on the YAML, re-run config import. Because import is upsert by name/UID, rolling forward to an older state is supported.

Drift detection

Re-run config export over the same target instance. If the resulting folder differs from what's committed, someone made a change in the UI. See Drift detection.

Troubleshooting

Symptom Likely cause Fix
Variable X not resolved An env var referenced in a connection YAML is unset export X=... or add to .env.
Datastore creation fails with connection 'X' not found The connection YAML wasn't part of the import (filtered with --include, or missing in the folder) Include connections, or pre-create them.
Re-import shows updates but the YAML didn't change A computed/timestamp field is being persisted The additional_metadata block sometimes carries server-side timestamps; that's expected if you're comparing exports done at different times.
403 Forbidden on connections Token has Member role Use a Manager token, or import with --include datastores,checks and pre-create connections out of band.
Re-export produces a non-empty git diff after no changes Different sort order, or a server upgrade changed an exported field File a bug; export should be deterministic.