Skip to content

Drift detection between environments

Goal

Detect when the live Qualytics configuration diverges from what's committed in Git. A typical compliance ask: "alert me when someone changes a quality check or datastore setting in the UI without going through the change management process." The CLI's deterministic config export makes this a one-line git diff.

Permissions

Step Endpoint Role Team permission
Export config (read everything) GET /api/connections, /api/datastores, /api/containers, /api/quality-checks Member Reporter on the datastore's team

Drift detection is read-only; Reporter is enough.

Prerequisites

  • The CLI is installed and authenticated.
  • A Git repository contains the committed configuration (the result of an earlier config export).
  • Optional: a notification destination (Slack webhook, email, PagerDuty) for when drift is found.

CLI workflow

graph LR
    Cron[Scheduled job] --> Pull[git pull]
    Pull --> Export[config export over committed folder]
    Export --> Diff[git diff]
    Diff -->|Diff present| Alert[Alert + open PR]
    Diff -->|Clean| Quiet[Exit 0]

Daily drift check

#!/usr/bin/env bash
set -euo pipefail

cd /opt/qualytics-config
git pull --quiet

# Re-export over the committed folder
qualytics config export \
    --datastore-id 42 \
    --output ./qualytics-config

# Anything different?
if git diff --quiet -- qualytics-config/; then
    echo "No drift."
    exit 0
fi

echo "Drift detected:"
git diff --stat -- qualytics-config/

# Surface the change however you alert
DIFF=$(git diff -- qualytics-config/ | head -200)
jq -n --arg text "Drift detected:\n\`\`\`\n$DIFF\n\`\`\`" '{text: $text}' \
  | curl -s -X POST -H 'Content-Type: application/json' \
         --data-binary @- \
         "$SLACK_WEBHOOK"

Run from cron

# Hourly during business hours
# Load credentials from a restricted file instead of inline:
#   echo 'export QUALYTICS_TOKEN=...' > /etc/qualytics-secrets && chmod 600 /etc/qualytics-secrets
0 9-17 * * 1-5 cd /opt/qualytics-config && . /etc/qualytics-secrets && QUALYTICS_NO_BANNER=1 ./drift-check.sh

Behind the scenes

CLI step Method Path Notes
config export (paginated reads of every resource type) GET /api/connections, /api/datastores/{id}, /api/containers?datastore_id={id}, /api/computed-fields, /api/quality-checks?datastore_id={id} Each resource is read and serialized to YAML in a deterministic order.

The CLI's export serializer sorts keys alphabetically, normalizes secrets to ${ENV_VAR} placeholders, and writes one resource per file. Re-running the export produces zero diff if nothing changed on the server, which is what makes Git-based drift detection reliable.

Python equivalent

import os
import subprocess
import sys

QUALYTICS_CONFIG_DIR = "/opt/qualytics-config"

# 1. Pull latest committed config
subprocess.run(["git", "pull", "--quiet"], cwd=QUALYTICS_CONFIG_DIR, check=True)

# 2. Re-export over the committed folder
env = {**os.environ,
       "QUALYTICS_URL":   "https://prod.qualytics.io",
       "QUALYTICS_TOKEN": os.environ["QUALYTICS_TOKEN"]}
subprocess.run(["qualytics", "config", "export",
                "--datastore-id", "42",
                "--output", "./qualytics-config"],
               cwd=QUALYTICS_CONFIG_DIR, env=env, check=True)

# 3. Compare
diff = subprocess.run(["git", "diff", "--exit-code", "--", "qualytics-config/"],
                      cwd=QUALYTICS_CONFIG_DIR)
if diff.returncode == 0:
    print("No drift.")
    sys.exit(0)

# 4. Drift found, surface it
diff_text = subprocess.check_output(
    ["git", "diff", "--", "qualytics-config/"], cwd=QUALYTICS_CONFIG_DIR, text=True
)
print("Drift detected:")
print(diff_text[:2000])
sys.exit(1)

Variations and advanced usage

Auto-open a PR with the drift

When drift is found, instead of just alerting, commit the new state to a branch and open a PR for review:

BRANCH="drift/$(date -u +%Y-%m-%dT%H%M)"
git checkout -b "$BRANCH"
git add qualytics-config/
git commit -m "drift: snapshot from $(date -u +%Y-%m-%dT%H:%M:%SZ)"
git push origin "$BRANCH"
gh pr create --title "Qualytics drift detected" \
             --body  "Drift snapshot from production. Review and decide whether to merge or revert in the UI."

Per-resource alerting

The git diff is by file, so you can grep for the resource type that drifted:

CHANGED=$(git diff --name-only -- qualytics-config/)
echo "$CHANGED" | grep -q connections/  && alert "Connection drift!"
echo "$CHANGED" | grep -q checks/       && alert "Quality check drift!"

Drift between two live environments

Compare Dev to Prod by exporting both and diffing the folders:

QUALYTICS_URL=https://dev.qualytics.io  QUALYTICS_TOKEN=$DEV_TOKEN  qualytics config export --datastore-id 1  --output ./dev/
QUALYTICS_URL=https://prod.qualytics.io QUALYTICS_TOKEN=$PROD_TOKEN qualytics config export --datastore-id 99 --output ./prod/

diff -ruN ./dev/datastores/<dev-name>/checks/ ./prod/datastores/<prod-name>/checks/

The directory names differ per datastore name, so map them before diffing if your naming differs across environments.

Troubleshooting

Symptom Likely cause Fix
Diff shows changes every run, even with no real change Server-side fields in additional_metadata (e.g., timestamps) are being persisted Filter those fields out before diffing, or compare only the resource bodies you care about.
"No drift" but UI clearly changed Wrong datastore ID in the export Confirm the datastore ID matches what was exported originally.
Drift check fails on first run after a Qualytics upgrade New fields appear in the export schema Re-baseline: commit the new export and start watching from there.
Slack alert truncates the diff Slack message size limit Send a link to the PR or to a paste service instead of the raw diff.